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1 What’s New 


1.1 Overview 


This document is the User Guide for Solarflare Enhanced PTP (sfptpd) which is an 
enhanced PTP daemon for use with Solarflare hardware timestamping network 
adapters. 


This issue of the user guide supports sfptpd from version 3.2.1 - 3.3.0. 


1.2 New features in 3.3.0.1007 


Best Master Clock Discriminator 


The bmc_discriminator option in the sfptpd configuration file is a means to 
disqualify PTP masters which advertise a time that differ by more than a specified 
threshold from a selected time source such as the selected NTP server. 


Refer to Best Master Clock Discriminator on page 89 for details of this feature. 


Epoch Guard 


sfptpd will now detect when a local NIC clock has been reset to the UTC epoch (01 
Jan 1970). This can occur, for example, following a NIC reset. 


Such an event will raise an alarm and, by default, prevent the affected NIC clock 
from altering other NIC clocks or the system clock. A further option can be enabled 
to immediately return the NIC clock to a coherent time. 


For further details, refer to Epoch Guard on page 90. 


UTC Offset Override 


A new option is added to the ptp_utc_valid_handling parameter in the sfptpd 
configuration file. 


The override option sets a fixed UTC offset (seconds) value and sfptpd will ignore the 
UTC offset value sent by the upstream master clock. The option is provided as a 
means of protecting clocks against misbehaving PTP master clocks. 


See ptp_utc_valid_handling on page 57. 
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Hybrid Network Mode Without Fallback 


A further option is added to the ptp_network_mode setting in the sfptpd 
configuration file to prevent fallback to multicast mode, maintaining sfptpd in 
hybrid mode. 


Refer to PTP Network Mode on page 91 for details. 


Freerun with System Clock 


An enhancement to the freerun synchronizaton mode allows the user to select the 
system clock as the Local Reference Clock. 


In combination with other options, this enables sfptpd to operate alongside chronyd 
for some use cases. 


See the Freerun Sync Module chapter for details. 


Chronyd 


sfptpd supports a limited use case with sfptpd running alongside chronyd. In such a 
configuration, chronyd will control/adjust the system clock whilst sfptpd can only 
read the system clock. 


For more information, refer to Chronyd on page 91. 


Clock Readonly 


The clock_readonly option is a list of clocks which sfptpd should never synchronize 
or adjust. 


This option can be used, for example, to prevent sfptpd from adjusting the system 
clock when running alongside chronyd. 


For more information, refer to Clock Readonly on page 92 for details. 


Systemd example 


sfptpd now includes and will install a system service UNIT file instead of the sysv- 
style initscript. A basic example configuration file allowing sfptpd to run alongside 
chronyd is also installed. 


For details of installed files, refer to Using the Binary RPM Package: SF-113122-LS on 
page 138. 


Configurable state path 


The state_path option allows the user to select the destination directory for sfptpd 
generated clock state and stats files. 


By default these files are maintained in the /var/lib/sfptpd directory as 
recommended by FHS 3.0 standards. 
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Delay-Request Intervals 


Previous sfptpd releases allowed a maximum Delay-Request interval of 4 (244 =16 
seconds) ignoring any larger value set by an upstream master clock. Version 
3.3.0.1007 sfptpd aligns with the IEEE-1588 default PTP specification which 
supports an interval of 5 (2*5=32 seconds). 


G) NOTE: An Delay-Request interval of 32 seconds is not recommended and may lead 
to poor synchronization performance. Ideally the Delay-Request interval will be at 
the same rate as received SYNC messages. 


Sync Instance Alarms 
Changes to sync instance alarm states are now logged in the sfptpd message log. 


e.g. 


<date> <time> notice: ptp1: alarms changed: none -> no-rx-timestamps 


1.3 New features in version 3.2.6 


VMware 


Support PTP synchronization in multiple Virtual Machines hosted on VMware ESXi. 
Refer to PTP on VMware ESXi on page 93 for details. 


Statistics 


Include time of minimum and maximum values within each time range. See the 
Monitoring on page 117 for more information. 


Sync Instance Selection 


Provide an additional option to identify the initial sync instance selected using the 
manual-startup mode of the selection_policy directive. 


Behaviour Changes 


no-delay-resps Alarm 


In previous releases, the no-delay-resps alarm would be triggered for every 
Delay-Response message not received or received too late - so triggers a timeout. 
Because missing the occassional Delay-Response message should not adversely 
affect the synchronization accuracy or performance, in the 3.2.6 release, the alarm 
will only trigger after 3 consecutive Delay-Response messages are not received. 


For a full description of behaviour changes and bug fixes, refer to the sfptpd Release 
Notes - ChangeLog section. 
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Documentation changes 


This issue of the user guide includes a TroubleShooting guide to assist users with 
common PTP issues. 
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Introduction 


This document describes Solarflare Enhanced PTP (sfptpd) support for Solarflare's 
Time Synchronization Server Adapters. These adapters support hardware time 
stamps of PTP packets, and can be deployed in networks where there is a 
requirement to support the IEEE 1588 Precision Time Protocol. 


The Solarflare sfptpd daemon is an implementation of the IEEE 1588-2008 Precision 
Time Protocol version 2. 


2.1 Supported PTP Profiles 


Solarflare sfptpd supports the following PTP profiles: 
e  JEEE-1588 (20008) Default profile 
e —_ Enterprise Profile 


sfptpd does not support the ITU -T G.8265 Telecoms PTP Profile and does not 
support unicast negotiation. 


2.2 Transport 


Solarflare sfptpd operates PTP over UDP/IPv4. The following delay methods are 
supported: 


e End-to-End (default) 
e Peer-to-Peer 


PTP over layer 2 (Ethernet) is not supported. 
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2.3 Definitions, acronyms and abbreviations 


1PPS 1 Pulse Per Second 


LACP Link Aggregation Control Protocol, part of IEEE-802.3ad 


LRC Local Reference Clock. The active clock to which all other clocks, ona 
PTP enabled server, are synchronized 


MTIE Maximum Time Interval Error 


NSM (Meinberg) NetSync Monitor 


NTP Network Time Protocol 


PF PCle Physical Function 


PHC PTP Hardware Clock 


PID Proportional, Integral, Differential filter 


PPB Parts Per Billion 


PPM Parts Per Million 


PTP Precision Time Protocol 


ptpd2 Animplementation of IEEE-1588-2008 (PTP version 2) 


sfptpd Solarflare Enhanced PTP daemon 
An implementation of IEEE-1588-2008 (PTP version 2) 


TLV Type, Length, Value - an element of a protocol or signaling message 


UUID Universally Unique Identifier 


VF PCle Virtual Function 


VLAN Virtual Local Area Network 


VM Virtual Machine 


2.4 Features 


Solarflare Enhanced PTP (sfptpd) supports the following features: 

e Hardware timestamps for received and transmitted PTP packets. 

e ~The ability to synchronize the high precision clock on multiple adapters. 
e — PTP clock roles: slave clock, master clock and boundary clock. 


e = PTP hybrid mode. 
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Unicast transmission of Delay_Request and Delay_Response messages to 
optimize use of network resources (Hybrid mode). 


PTP Synchronization mode. 

PPS Synchronization mode. 

NTP fallback mode. 

PTP multiple masters support 

Filtering of outlier values received from master clock sources. 

PTP over VLAN interfaces. 

PTP over active/backup and LACP bonded interfaces. 

Standards based, application-independent reporting of clock data. 
Remote monitoring/reporting of PTP alarm states and events. 

PTP using third party adapters. 


PTP in virtualized environments (VMware ESXi). 


2.5 Supported adapters 


Solarflare Adapters 


Solarflare XtremeScale™ X2 series adapters 
Solarflare XtremeScale™ SFN8000 series adapters. 
Solarflare Flareon™ SFN7000 series adapters. 


HP 570FLB FlexibleLOM and HP 570M Mezzanine adapters. 


An adapter must have a PTP/HW timestamping activation key installed. This key is 


pre-installed on ‘PLUS’ adapters and on the SFN7322F adapter. 


Non-Solarflare Adapters 


@ 


From version 3.2.1 sfptpd can recover hardware timestamps for PTP packets from 


third party adapters. sfptpd will also discipline the clocks on third party adapters 
that support the PHC API. 


This is an experimental feature where testing has shown that synchronization 


accuracy/performance in mixed adapter systems is best on Solarflare adapters. 


Solarflare have conducted limited testing with a range of third party adapters - the 
list of tested models can be found in the sfptpd release notes. 


NOTE: Implementation of the PHC API is not uniform across all non-Solarflare 
adapters, therefore this is a best effort feature. 
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Timestamping ports 


Solarflare PTP network adapters support hardware timestamping: 


e ~—- Solarflare XtremeScale™ X2 Series: 


- Hardware timestamping of all transmitted and received packets on all 
adapter ports. 


e ~—- Solarflare XtremeScale™ SFN8000: 


- Hardware timestamping of all transmitted and received packets on all 
adapter ports. 


e = Solarflare Flareon™ SFN7000: 

- Hardware timestamping of all received packets on all adapter ports. 
e HP branded adapters: 

- Hardware timestamping of all received packets on all adapter ports. 


The adapter timestamp function is compliant with the IEEE 1588-2008 (PTP 
version 2) specifications. 
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3.1 Supported OS versions 


Solarflare Enhanced PTP is supported on the following OS/kernels versions: 


Table 1: OS/Kernel Support 


OS Version Notes 

Red Hat Enterprise Linux 6.4 - 7.4 RHEL6 built-in Solarflare drivers 
may not support SFN7000, 
SFN8000 and X2 series 
adapters. 


Red Hat Messaging Realtime and Grid 2.4, 2.5 


SuSE Linux Enterprise Server 11 sp2,sp3,sp4 Built-in Solarflare drivers may 
not support SFN7000, SFN8000 
and X2 series adapters. 


SuSE Linux Enterprise Realtime Extension 11 


SuSE Linux Enterprise Server 12 sp1, sp2 


Linux kernels 2.6.32 - 4.11 


Canonical Ubuntu Server LTS 14.04, 16.04 


Canonical Ubuntu Server 16.10, 17.04, 18.04 


Debian 7 “Wheezy” 


Debian 8 “Jessie” 


Debian 9 “Stretch” 


VMware ESXi 6.0 - 6.7 
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3.2 Solarflare PTP network adapters 


Solarflare time synchronization adapters generate hardware timestamps for PTP 
packets in support of a network precision time protocol deployment, and in 
accordance with the IEEE 1588-2008 specifications. With hardware precision and 
performance, the PTP adapters facilitate PTP slave servers to accurately synchronize 
internal clocks to multiple network master clocks, or to serve as either a master 
clock or a boundary clock. 


These adapters contain a dedicated time stamping unit which is driven from a high 
precision oscillator. Receipt or transmission of a PTP formatted packet triggers the 
generation of an accurate hardware timestamp which is passed by the adapter to 
the network device driver. The adapter also enables a PTP stack running on the host 
server to discipline the adapter's precision oscillator (both absolute time and clock 
rate). 


The PTP stack running on the host server will synchronize the adapter clock and host 
real time clocks to a remote time source. 


XtremeScale X2 series single and dual-port 10/25/40/50/100GbE SFP28/ 
QSFP adapters 


XtremeScale™ SFN8000 series dual-port 1OGbE SFP+ and 40GbE QSFP+ 
adapters 


The X2 and SFN8000 series adapters combine ultra low latency with precision time 
synchronization and hardware timestamping of all received network packets on 
either physical port of the adapter. 


Ports 


PTP packets can be received on any adapter port and ports can be configured in an 
active/backup failover configuration. LACP bonding is also supported. Hardware 
timestamping is done on all ports. 


PTP Activation Key 
Plus adapters, are supplied as a factory-ready PTP adapter. 


Other X2 series and SFN8000 series adapters can be upgraded with a Solarflare 
AppFlex™ Technology PTP/HW activation key to support PTP and hardware 
timestamping of all received packets. 


For more details of the AppFlex Technology activation keys refer to the Solarflare 
Server Adapter User Guide (SF-103837-CD). 
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Flareon™ SFN7000 series dual-port 1OGbE SFP+ and 40GbE QSFP+ 
adapters 


The SFN7000 series adapters combine ultra low latency with precision time 
synchronization and hardware timestamping of all received network packets on 
either physical port of the adapter. 


Ports 


PTP packets can be received on any adapter port and ports can be configured in an 
active/backup failover configuration. LACP bonding is supported. Hardware 
timestamping is done on all ports. 


PTP Activation Key 
The SFN7322F adapter is supplied as a factory-ready PTP adapter. . 


Other SFN7000 series adapters can be upgraded with a Solarflare AppFlex™ 
Technology PTP/HW activation key to support PTP and hardware timestamping of all 
received packets. 


For more details of the AppFlex Technology activation keys refer to the Solarflare 
Server Adapter User Guide (SF-103837-CD). 


PPS Support 


A 1PPS bracket kit and cable assembly providing PPS input/output connections can 
be fitted to the X2 series (not available for the X2541), SFN8000 or SFN7000 series 
adapters: 


e  SFP+, SFP28 adapters require the SOLR-PPS-DP10G bracket kit 
e = =QSFP+, QSFP28 adapters require the SOLR-PPS-DP40G bracket kit. 


Customers interested in the optional PPS kit should contact their Solarflare sales 
channel. 


HP-branded server adapters 


The HP570FLB and HP570M dual-port 10G Ethernet adapters are server adapters for 
the HP c-Class ProLiant Gen8 BladeSystem. These adapters are functionally 
equivalent to the Solarflare SFN7122F adapter and can support hardware 
timestamping on either port when the PTP/HW timestamping activation key is 
installed. 
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3.3 Time synchronization features 


Solarflare PTP network adapters have the following time synchronization features: 


e = Ability to maintain synchronization of the system clock typically within 200ns 
offset from a network master clock. The accuracy obtained is dependent on the 
correct operation of the upstream timesource, however, the PTP slave adapter 
clock can be within 50ns offset from the PTP master. 


e Stratum 3 compliant oscillator; Oscillator drift 0.37 PPM per day (c. 32ms/day); 
oscillator accuracy < 4.6PPM over 20 years. 


e Ability to capture a hardware timestamp as selected frames enter/leave the 
Ethernet MAC. Time stamping for packets formatted according to IEEE 1588- 
2008 (PTP version 2). 


e Hardware timestamps exposed to Linux via the standard SO_TIMESTAMPING 
socket API on kernels 2.6.30 later. 


e Ability to discipline the network adapter’s high precision oscillator in response 
to PTP timing information. 


e¢ Support PTP packets over bonded interfaces in an active/standby configuration 
or LACP bonds. 


e Support PTP packets over 802.1Q VLAN interfaces. 


3.4 Adapter Driver/Firmware Support 


This section identifies the minimum software components required to support time 
synchronization server adapters. 


To identify Solarflare adapters in a server, run the following command: 

# lspci -vvv -d 1924: 

The following command typically extracts the adapter name from the above output: 
# lspci -vvv -d 1924: | egrep 'Product|Subsystem' 


To identify the Solarflare net driver and firmware versions used by the Solarflare 
adapter run the following command where N is the Solarflare interface: 


# ethtool -i eth<N> . 


Adapter Linux net driver Controller firmware 
X2 Series Any Any 

SFN8542 Any Any 

SFN8522 Any Any 

SFN7142Q v4.1.0.6734 v4.1.1.1022 
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Adapter Linux net driver Controller firmware 
SFN7322F v4.0.2.6628 v4.0.6.6689 
SFN7122F 
SFN7002F 
HP 570FLB v4.0.2.6628 v4.0.7.6711 
HP 570M 
SFN6322F v3.3.0.6246 V3.3.0.6247 


OS ‘in-tree’ Driver 


The ‘in-tree’ driver included with many Linux distributions does not support PTP 
features required by sfptpd. To identify if the adapter is using an ‘in-tree’ driver run 
the following ethtool command: 

# ethtool -i <interface> 


version: 4.0 * 
firmware-version: 3.3.0.6298 


* The ‘in-tree’ driver is version 4.0 or 4.1. 


CAUTION: The sfptpd daemon must be terminated before upgrading the adapter 
l \ firmware. 


Network driver 


The adapter driver is distributed as a standalone RPM (source and DKMS) or with 
the OpenOnload/EnterpriseOnload distributions (listed below). 


¢  OpenOnload from version 201310-u1. 
e — EnterpriseOnload from version 3.0.0.2. 
The network driver exposes a number of features of the adapter including: 


e Hardware time stamping of PTP formatted packets. 


Starting in Linux kernels 2.6.30, support for hardware time stamping of 
network packets on TX and RX is formalized and integrated via the socket 
option SO_TIMESTAMPING. Details of this interface can be found at http:// 
Ixr.linux.no/linux/Documentation/networking/timestamping.txt. Before an 
application can receive timestamps on a socket it must first issue the IOCTL 
SIOCSHWTSTAMP to register both the types of packets it wants to receive 
timestamps on and the format of timestamps it wishes to receive. 
SIOCSHWTSTAMP is not required by applications using Onload. 


-  X2 series adapters can hardware time stamp all transmitted and received 
packets. 
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- SFN8000 series adapters can hardware time stamp all transmitted and 
received packets. 


- SFN7000 series adapters can hardware time stamp all received packets. 
- HP-branded adapters can hardware time stamp all received packets. 


- The SFN6322F hardware time stamping is limited to PTP formatted 
packets. 


e Access to the control of the precision oscillator. 


The adapters contain a precision clock with drift rated to < 1 PPM per year. The 
driver allows the absolute time and frequency of this clock to be controlled via 
the Linux PHC subsystem or the proprietary IOCTL interface. The proprietary 
interface is used by the Solarflare supplied sfptpd stack to run the PTP 
protocol using the precision oscillator on the adapter. 


3.5 Synchronization model 


The sfptpd Solarflare Enhanced PTP daemon synchronizes local clocks, including 
the system clock, to a single time source: 


e sfptpd can process multiple time sources including: 
- one or more remote PTP master clocks 
- a 1PPS signal, with an external NTP server providing time of day 
- an external NTP server 
- a local Solarflare adapter clock. 


Each time source is provided by an instance of a sync module. For more 
information, see Sync modules on page 15. 


e sfptpd selects a time source to use. 


By default, sfptpd automatically selects the best performing time source. For 
more details, see Sync module selection on page 19. 


¢ —sfptpd synchronizes a Local Reference Clock (LRC) to the selected time source. 


sfptpd designates one local clock as the Local Reference Clock (LRC), and 
synchronizes this to the selected external time source. 


The LRC used will depend on the sync module selected as the time source, but 
the LRC is typically the precision clock on a Solarflare PTP adapter. See the 
descriptions of sync modules in Sync modules on page 15. 


e — sfptpd synchronizes each clock, including the system clock, to the LRC. 


sfptpd runs a clock servo for each additional clock in a server i.e. for each 
additional timestamping adapter and for the system clock. sfptpd will 
measure the difference between each clock and the LRC, and synchronize all 
clocks to the LRC. 
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Figure 1 illustrates the synchronization options supported by sfptpd. 


sfptpd 
Synchronization 
Model 


system time PTP Slave 


PTP Mode Free-run 
Remote Master ; : 


Figure 1: sfptpd synchronization model 


3.6 Sync modules 


The following sections provide an overview of the available sync modules: 
e —PTP sync module on page 16 

e — PPS sync module on page 17 

e NTP sync module on page 17 

e Freerun sync module on page 18. 


Each instance of a sync module represents a candidate time source to sfptpd. From 
the group of candidates, one instance is selected as the active time source. This 
process is described in Sync module selection on page 19. 
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PTP sync module 


Using a PTP sync module as the time source, sfptpd will synchronize the adapter 
clock and system clock to an upstream PTP master clock: 


e When the PTP sync module uses an interface on a Solarflare PTP-enabled 
adapter, the adapter clock is selected as the LRC. 


sfptpd synchronizes the LRC to the PTP master, and keeps all other clocks 
(including the system clock) synchronized with the LRC. 


system time 


PTP Master 
PTP 


Figure 2: PTP sync module 


e When the PTP sync module uses an interface on a non-PTP enabled adapter, 
sfptpd will discipline the host system clock time and only software 
timestamping is available. 


e A third party adapter supporting the PHC API can also be selected as the LRC. 
Refer to release notes for a list of viable adapters. 


Refer to PTP Sync Module on page 52 for more details. 
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PPS sync module 


When sfptpd selects a PPS sync module as the time source, it periodically receives 
time-of-day from an external NTP server, and then maintains clock synchronization 
using a received 1PPS signal: 


e If the PPS sync module is using an interface on a Solarflare PTP-enabled 
adapter, the adapter clock is selected as the LRC. 


sfptpd periodically polls an NTP client for time-of-day. Thereafter sfptpd 
synchronizes the LRC to the 1PPS pulse, and keeps all other clocks (including 
the system clock) synchronized with the LRC. 


PTP Slave 
>> -_ = 


Master Clock 


Figure 3: PPS sync module 


The NTP client is used to provide the time-of-day to the slave system clock. 


G) NOTE: This sync module does not use PTP, and so does not send or process any PTP 
messages. 


For more details see PPS Sync Module on page 71. 


NTP sync module 


When sfptpd selects an NTP sync module as the time source, it synchronizes the 
adapter and system clocks to an external NTP server. 


The system clock is selected as the LRC. 


An local NTP client is used to synchronize the system clock with an external NTP 
server. sfptpd keeps all other clocks synchronized with the system clock, so 
hardware timestamps (of non-PTP packets) received from the adapter can be 
compared with system time. 


NTP Server 


SFN7x22F 


network packets 


Figure 4: NTP sync module 


An accurate and stable NTP server should be selected when using this mode. 
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G) NOTE: This sync module does not use PTP, and so does not send or process any PTP 
messages. 


For more details see NTP Sync Module on page 74. 


Freerun sync module 


When sfptpd selects a Freerun sync module as the time source, it synchronizes the 
adapter and system clocks to a local time source. 


Either the system clock or the precision clock on a Solarflare PTP adapter can be 
selected as the LRC. 


If the LRC is an adapter clock, at startup, sfptpd sets the time of the LRC from the 
system clock. Thereafter the LRC runs free, and sfptpd keeps all other clocks 
(including the system clock) synchronized with the LRC. 


The system is not being synchronized to a remote time source. 


This sync module can be useful as a backup or failover, for use when other more 
reliable time sources become unavailable. 


system time 


Figure 5: sync_mode freerun with freerun_mode nic 


G) NOTE: This sync module does not use PTP, and so does not send or process any PTP 
messages. 


For more details of the Freerun sync module, see Freerun Sync Module on page 78. 
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3.7 Sync module selection 


This section describes how a single sync module instance is selected as the time 
source. By default selection is automatic, Manual selection is also an option. 


Automatic selection 


By default, sfptpd automatically selects the time source to use. The set of all 
configured sync instances represents a list of candidates available for selection. 


A sync instance in the SLAVE state is considered before another instance in the 
LISTENING state. 


A candidate is automatically selected using the following criteria, in descending 
order of priority: 


1 = State 


A sync instance in the SLAVE state is considered before another in the 
LISTENING state. 


2 NoAlarms 


A sync instance having no PTP alarms present is considered before others 
where alarms are active. 


3 Instance priority 


The value of the priority configuration option for the instance. Smaller values 
have higher priority. By default, all instances have priority 128, and so this 
criterion is ignored 


G) NOTE: This is not related to the IEEE-1588 specification priority 1 and priority 2 
properties of a PTP master clock. 


4 Clock class 


Locked clocks are considered ahead those which are in holdover or are free- 
running. 


5 Accuracy 
This is the sum of: 


a) Clock accuracy 
Estimate of error between clock and primary reference source. 


b) Sync module accuracy 


Estimate of error implied by the synchronization mechanism that would 
be used to sync to the remote clock, such as NTP with software 
timestamping vs. PTP with hardware timestamping. 


6 Allan variance 
Estimate of stability of clock. 
7 Identity of clock 


If all other comparisons fail, the identity of the clock is used as a tie-breaker. 
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The source of the clock class and accuracy varies: 


e For the PTP sync module, the clock class and clock accuracy are values 
conveyed in PTP messages from the upstream PTP master clock. 


e For the PPS sync module, the clock class and clock accuracy have default values 
but can also be set by the user. 


e For the NTP sync module, the clock class and clock accuracy come from the 
remote clock. 


e For the Freerun sync module, the clock class and clock accuracy use default 
values. 


Change the Automatic Selection Order 


The order of selection criteria can be changed to suit a specific network or system 
requirement. 


To change the default ordering of selection criteria, the user should change the 
order of criteria as they appear with the following configuration file option: 


# Specify an alternative ordering of rules for the selection policy. 


selection_policy_ rules manual state no-alarms user-priority clock-class 
total-accuracy allan-variance steps-removed 


See the supplied /config/default .cfqg file for details. 


This feature allows a hybrid manual-automatic selection priority when the ‘manual’ 
option is moved to a different position. 


Selection Holdoff Interval 


When using the automatic selection policy, the selection_holdoff_interval (default 
10 seconds) can be used to delay switchover to another sync_instance or to prevent 
rapid switching between instances having similar characteristics. 


The selection_holdoff_interval can be set in the sfptpd config file. 


Manual selection 


To use manual selection, set the selection_policy configuration option to 
manual, with a second parameter giving the name of the initial sync module 
instance to use. See Configuration options on page 33. 


Use the sfptpdct1 utility program to change the instance selected as the master. 
This allows the user to switch timesources simultaneously across an entire network, 
thus ensuring that all servers use a common timesource. See Daemon control 
mechanism on page 86. 
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3.8 Sync Module Accuracy 


The following table identifies the notional accuracy value for each sync_module 


type. 
Sync_module Accuracy 
Freerun 0 
NTP 10 us 
PPS 50 ns 
PTP_HW 50 ns 
PTP_SW 50 ms 
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Procedure to install components and run sfptpd. 
Installation on page 22 

Download and install sfptpd on page 25 
Pre-Start Checks on page 27 

Run sfptpd on page 28 


4.1 Installation 


Follow steps 1-6 to install the Solarflare Adapter, check driver and firmware versions 
and check that required PTP activation key is loaded. 


Step 1: Verify Solarflare adapter driver and firmware versions 


@ 


To check the Solarflare adapter driver and firmware versions use the Linux ethtool 
command e.g. 


# ethtool -i eth<N> 


NOTE: The in-tree Solarflare drivers that are supplied with Linux distributions do 
not support PTP. The in-tree driver will be version 4.0 or 4.1, for example, this is an 
in-tree driver that does not support PTP: 

# ethtool -i eth<N> 

sfc 4.0 


Refer to Chapter 3 on page 9 for driver and firmware information. 


Step 2: Install the network adapter 


Complete instructions for the deployment and installation of the network adapter 
can be found in the Solarflare Server Adapter User Guide (SF-103837-CD). 


Step 3: Install the network driver 


The network adapter driver is available from https://support.solarflare.com/ in the 
following formats: 


e DKMS (part number SF-104979-LS) 
e Source RPM (part number SF-103848-LS). 


e The OpenOnload® and EnterpriseOnload® distributions also include a version 
of the adapter driver. 
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Complete instructions for installing and loading the driver can be found in the 
Solarflare Server Adapter User Guide (SF-103837-CD). 


Step 4: Update the adapter firmware 


ZN 
@ 


The adapter firmware only needs to be updated if it does not meet the minimum 
firmware version required to support PTP and HW timestamping. 


CAUTION: The sfptpd daemon must be terminated before upgrading the adapter 
firmware. Following firmware upgrade the adapter driver should be reloaded 
before starting sfptpd. 


NOTE: The Solarflare Utilities RPM for Linux contains the Solarflare Boot Manager 
(sfboot), a flash firmware update utility (sfupdate) and an AppFlex activation key 
upgrade utility (sfkey). 


The RPM package is available as a 32bit binary and 64bit binary: 
e SF-105095-LS is a 32bit binary 
e SF-107601-LS is a 64bit binary 


1 Download the sfutilities package from https://support.solarflare.com/. 
2 Unzip the file to reveal the binary RPM 
3 Install the RPM e.g. 

# rpm -Uvh sfutils-<version>.rpm 


4 Identify the current firmware version on the adapter. 
# sfupdate 

5 Replace the adapter firmware with the version in this sfupdate. 
# sfupdate --write 


Full instructions on using sfupdate, sfboot and sfkey can be found in the 
Solarflare Network Adapter User Guide (SF-103837-CD). 


Step 5: Ensure PTP activation keys are installed 


An adapter must have an AppFlex PTP activation key installed before it can be used 
for PTP. The key is pre-installed on certain adapters. Use the sfkey utility to list keys, 
and to install a key if necessary. 


1 _— Display the installed keys for all adapters: 


# sfkey --all --report 
eth2,eth3: 712200201082133128200003 (Flareon) 
Product name Solarflare SFN7122F SFP+ Server Adapter 
Installed keys Onload, SolarCapture Pro 
If the Installed keys do not include a PTP or a Plus key, install a PTP 
activation key: 
Copy the activation key data to a .t xt file on the target server. All keys can be 


in the same key file and the file applied on multiple servers. The following 
example uses a activation key file called keys .txt created on the local server. 
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# sfkey --adapter=eth2 --install keys.txt 
Reading keys... 


Writing all keys to eth2... 

eth2: 712200201082133128200003 (Flareon) 
Product name Solarflare SFN7122F SFP+ Server Adapter 
Installed keys Onload, PTP, SolarCapture Pro 
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4.2 Download and install sfptpd 


The Solarflare Enhanced PTP, sfptpd, is a PTP daemon adapted by Solarflare to work 
with Solarflare time synchronization server adapters. The sfptpd daemon is an 
implementation of IEEE-1588-2008 (PTP version 2). 


G) NOTE: PTP Version 2 is not compatible with PTP Version 1. They use different 
message sizes and format. Version 1 messages received by sfptpd will be ignored. 


The sfptpd package is available packaged as a tarball or as an RPM, in 64bit binary 
format: 


e SF-108910-LS is a 64bit binary tarball 
e SF-113122-LS is a 64bit binary RPM 
Download the required sfptpd package from https://support.solarflare.com/. 


Install tarball 


To install using a tarball package: 
1 Unpack the compressed file: 


# tar -zxvf SF-108910-LS-<version>.tgz 


sfptpd-3.3.0.1007.x86_64/ 
sfptpd-3.3.0.1007.x86_64/sfptpd 
sfptpd-3.3.0.1007.x86_64/examples/ 
sfptpd-3.3.0.1007.x86_64/examples/README.sfptpdctl 
sfptpd-3.3.0.1007.x86_64/examples/monitoring console.py 
sfptpd-3.3.0.1007.x86_64/examples/sfptpdctl.c 
sfptpd-3.3.0.1007.x86_64/examples/sfptpdctl.py 
sfptpd-3.3.0.1007.x86_64/examples/sfptpd_json_parse.html 
sfptpd-3.3.0.1007.x86_64/examples/Makefile.sfptpdctl 
sfptpd-3.3.0.1007.x86_64/examples/sfptpd_stats_collectd.py 
sfptpd-3.3.0.1007.x86_64/PTPD2_COPYRIGHT 
sfptpd-3.3.0.1007.x86_64/NTP_COPYRIGHT. html 
sfptpd-3.3.0.1007.x86_64/init.d/ 
sfptpd-3.3.0.1007.x86_64/init.d/sfptpd 
sfptpd-3.3.0.1007.x86_64/sfptpdctl 
sfptpd-3.3.0.1007.x86_64/LICENSE 
sfptpd-3.3.0.1007.x86_64/config/ 
sfptpd-3.3.0.1007.x86_64/config/ptp_master_ntp.cfg 
sfptpd-3.3.0.1007.x86_64/config/ptp_slave_multiple.cfg 
sfptpd-3.3.0.1007.x86_64/config/default-systemd.cfg 
sfptpd-3.3.0.1007.x86_64/config/freerun. cfg 
sfptpd-3.3.0.1007.x86_64/config/ptp_slave_ntp_fallback.cfg 
sfptpd-3.3.0.1007.x86_64/config/ptp_slave.cfg 
sfptpd-3.3.0.1007.x86_64/config/ntp.cfg 
sfptpd-3.3.0.1007.x86_64/config/ptp_domain_bridge.cfg 
sfptpd-3.3.0.1007.x86_64/config/many_instances.cfg 
sfptpd-3.3.0.1007.x86_64/config/pps_slave.cfg 
sfptpd-3.3.0.1007.x86_64/config/ptp_master_freerun.cfg 
sfptpd-3.3.0.1007.x86_64/config/ptp_boundary.cfg 
sfptpd-3.3.0.1007.x86_64/config/default.cfg 
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2 Startup components: 


e Aninit.dscript is supplied to allow sfptpd to be started using standard Linux 
service: start, stop, restart, status commands. 


The init.d script should be copied to /etc/rc.d/init.d. 
e Anexample systemd UNIT file can be found in Appendix B on page 134. 


e Default configuration files for all modes are located in the config subdirectory 
of the install directory. 


Install REM 
To install using an RPM package: 


1 Unpack the RPM package: 


tar -zxvf SF-113122-LS- 
<version>_Solarflare_Enhanced_PTP_Daemon_sfptpd_- 
_64_bit_binary_RPM.tgz 


2 Install the binary RPM: 


rpm -ivh sfptpd-<version>.x86_64.rpm 
Preparing JAHRE EH [108% | 


Updating / installing... 
1:sfptpd-3.3.0.1007-1 +HREHHHHHHEHHHHHHH HHH HHH [100% | 


Files installed 
# rpm -ql sfptpd-3.3.0.1007-1.x86_64 


/etc/sfptpd. conf 

/usr/lib/systemd/system/sfptpd.service 

/usr/sbin/sfptpd 

/usr/sbin/sfptpdctl 

/usr/share/doc/packages/sfptpd 
/usr/share/doc/packages/sfptpd/LICENSE 
/usr/share/doc/packages/sfptpd/NTP_COPYRIGHT .html 
/usr/share/doc/packages/sfptpd/PTPD2_COPYRIGHT 
/usr/share/doc/packages/sfptpd/config 
/usr/share/doc/packages/sfptpd/config/default-systemd.cfg 
/usr/share/doc/packages/sfptpd/config/default.cfg 
/usr/share/doc/packages/sfptpd/config/freerun. cfg 
/usr/share/doc/packages/sfptpd/config/many_instances.cfg 
/usr/share/doc/packages/sfptpd/config/ntp.cfg 
/usr/share/doc/packages/sfptpd/config/pps_slave.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_boundary.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_domain_bridge.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_master_freerun.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_master_ntp.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_slave.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_slave_multiple.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_slave_ntp_fallback.cfg 
/usr/share/doc/packages/sfptpd/examples 
/usr/share/doc/packages/sfptpd/examples/Makefile.sfptpdctl 
/usr/share/doc/packages/sfptpd/examples/README .sfptpdctl 
/usr/share/doc/packages/sfptpd/examples/init.d 
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/usr/share/doc/packages/sfptpd/examples/init.d/sfptpd 
/usr/share/doc/packages/sfptpd/examples/monitoring console. py 
/usr/share/doc/packages/sfptpd/examples/sfptpd_json_parse. html 
/usr/share/doc/packages/sfptpd/examples/sfptpd_stats_collectd.py 
/usr/share/doc/packages/sfptpd/examples/sfptpdctl.c 
/usr/share/doc/packages/sfptpd/examples/sfptpdctl.py 
/usr/share/doc/packages/sfptpd/examples/systemd 
/usr/share/doc/packages/sfptpd/examples/systemd/sfptpd.service 


Startup Components: 


e The sfptpd executable will be installed into root’s path. 


e =A default systemd UNIT file is installed at: 
/usr/lib/systemd/system/sfptpd. service 

e = An init.d script is supplied in the /examples directory and can be copied to 
/etc/rc.d/init. 


4.3 Pre-Start Checks 


NTP 


The NTP service should only be running when using an sfptpd mode that requires 
NTP as a timesource or fallback timesource e.g. NTP fallback mode or NTP/PPS 
mode. 


If NTP is not required by the sfptpd mode, the service should be terminated before 
starting the sfptpd daemon. This is to prevent NTP from changing the system clock. 
sfptpd will synchronize the system clock. 


# service ntpd stop 
# systemctl status ntpd 


On startup sfptpd stats log, message _log output will warn the user if the NTP 
service is running. 


Chronyd, systemd-timesyncd, systemd-timedated 


All of these applications will periodically adjust the system clock. Ensure that these 
services are not running alongside sfptpd. 


If chronyd is required to run alongside sfptpd, refer to Chronyd on page 91. 


IPTables 


Users must ensure that no rule exists in iptables or firewalld that will prevent PTP 
packets from reaching the slave sfptpd process. The following command is a useful 
starting point to examine iptables rules. 


# iptables -L 
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The iptables service can be temporarily disabled to identify if it is responsible for 
blocking PTP traffic. 


# service iptables stop 


# systemctl stop firewalld 


PTP Domain 


A PTP slave must be in the same PTP Domain as the upstream master clock. 


PTP messages received by sfptpd that are not from the configured domain will be 
ignored. The default domain value in sfptpd config files is 0 (zero). 


The domain value should be configured to match the PTP domain of the upstream 
master clock. Configure the domain with the ptp_domain parameter in the sfptpd 
config file. 


Reverse Path Filters 


Reverse path filters, if enabled, can prevent PTP packets from being delivered to the 
sfptpd process. 


If packets are arriving at the PTP interface, but the following message is observed 
periodically in the sfptpd stats file: 


Failed to receive Announce message in 12 seconds 
Disable the rp_filter on the PTP interface - and (possibly required) all interfaces: 
# echo @ > /proc/sys/net/ipv4/conf/<interface>/rp_filter 
# echo @ > /proc/sys/net/ipv4/conf/all/rp_filter 


Also see the Failed to Receive Announce Messages on page 144 section. 


4.4 Run sfptpd 


To run sfptpd: 


1 Ensure that the timestamping port of the adapter is configured with an IP 
address suitable for the network configuration. 


2 Perform Pre-Start Checks on page 27 - especially the PTP domain. 
3 Start sfptpd using one of the supplied configuration files. 

- Run sfptpd as a PTP slave on page 29 

- Run sfptpd in freerun mode on page 29 


- Run sfptpd in NTP/PPS mode on page 29. 


G) NOTE: sfptpd can take 15-30 minutes to initially stabilize the times on the slave 
machines. 
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Run sfptpd as a PTP slave 
As a PTP slave, sfptpd is receiving and sending PTP messages to an upstream PTP 
master clock. 
For this mode, use the default ptp_slave.cfg file: 


e from tarball package: 


cd <install_dir> 
./sfptpd -i <interface> -f config/ptp_slave.cfg 


e from RPM package: 


cd /usr/share/doc/packages/sfptpd/ 
sfptpd -i <interface> -f config/ptp_slave.cfg 


where <interface> is the identifier of the timestamping port on the adapter. 


Run sfptpd in freerun mode 


In Freerun mode, sfptpd will synchronize all clocks in the server, including the 
system clock, to the high precision oscillator on the selected PTP adapter. 


For this mode, use the default freerun.cfg file: 


e from the tarball package: 


cd <install_dir> 
./sfptpd -i <interface> -f config/freerun.cfg 


e from the binary RPM package: 


cd /usr/share/doc/packages/sfptpd/ 
sfptpd -i <interface> -f config/freerun.cfg 


where <interface> is the identifier of the timestamping port on the adapter. 


Run sfptpd in NTP/PPS mode 


In NTP/PPS mode, sfptpd will get time-of-day from NTP, then use the 1PPS signal to 
closely synchronize the adapter clock. All other clocks, including the system clock, 
are synchronized to the adapter clock. 


NOTE: It is a requirement to setup NTP symmetric authentication to allow sfptpd to 
control the local NTP client. 


G) NOTE: Only a single NTP sync_module is required. Do not create multiple NTP sync 
modules. 


For this mode, use the default pps_slave.cfg file: 


e from the tarball package: 


cd <install_dir> 
./sfptpd -i <interface> -f config/pps_slave.cfg 


e from the binary RPM package: 


cd /usr/share/doc/packages/sfptpd/ 
sfptpd -i <interface> -f config/pps_slave.cfg 
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where <interface> is the identifier of the timestamping port on the adapter. 


Issue 8 © Copyright 2020 Xilinx, Inc 30 


Solarflare Enhanced PTP User Guide 


SOLARFLARE® 


A XILINX COMPANY 


5 User Interface 


5.1 Command line options 


The configuration settings for sfptpd are enabled, set or disabled in a configuration 
file. 


The path to the configuration file is specified on the sfptpd command line. For 
example, to use the default ptp_slave.cfg file in the config sub-directory: 


./sfptpd -i eth<N> -f config/ptp_slave.cfg 


sfptpd also supports a limited number of command line options which override the 
equivalent config file options. Use the sfptpd -h command to identify supported 
command line options. 


5.2 Configuration files 


Comments 


Blank lines and comment lines, beginning with a # symbol, are ignored. 


Structure of configuration files 
sfptpd configuration files use a simple INI file format. Options are grouped under 
labelled sections enclosed in square brackets. For example: 
[general] 


Sync modules are created in the [general] section. 


Add a single sync_module option per type of sync module that is required. The first 
parameter identifies the type of sync module, and following parameters provide the 
name of each instance of the module. 


The following example creates two PTP sync module instances, named ptp1 and 
ptp2: 
sync_module ptp  ptp1 _ ptp2 

| | | 


type | | 
name | 
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A sync module can have generic options that apply to all instances of the 
sync_module type, and further instance-specific options that apply only to a specific 
instance of the sync_module: 


e Generic options are set in a section of the configuration file labeled with the 
type of sync module. For example: 


[ptp] 


e —_Instance-specific options are set in a section of the configuration file labeled 
with the instance name. For example: 


[ptp1] 
Example of structure 
Below is an example of the structure in a configuration file: 
[general] 
# specify instances of the PTP sync module named ptp1 and ptp2 


sync_module ptp ptp1 ptp2 


# other generic sfptpd options 


[ptp] 
# generic PTP options, applied to all ptp instances 


[ptp1] 
# instance-specific PTP options, applied to the instance named ptp1 


[ptp2] 
# instance-specific PTP options, applied to the instance named ptp2 


Supplied configuration files 


The sfptpd distribution provides default configuration files for all supported modes 
in the config sub-directory: 
e Default options are set within each config file. For example: 
ptp_domain @ 
Some options will require changing the values to match the specific PTP 
network. 
e Additional options are commented out. For example: 
#ptp_announce_interval 1 


These can be selected by un-commenting the option line and, if required, 
entering a different value for the option: 


ptp_announce_interval 3 


Some of these files contain generic options, and are described in Example 
configuration files on page 37. The remaining files in the config directory give 
examples of how to use the different sync modules. For descriptions of these, see 
the chapters describing the sync modules. 
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Creating additional configuration files 


The user is free to create additional configuration files and store these anywhere on 
the local server. Configuration files can have any name, and by convention have a 
. cfg file extension. The path to the file is given on the sfptpd command line. 


5.3 Configuration options 


Table 2 lists configuration options for sfptpd. These are set in a section of the 
configuration file labeled [general] 


Option 


sync_module 


Table 2: Configuration options for sfptpd 


Parameters 


freerun|ptp|pps|ntp 
[<instance-name(s)>] 


AN 


Description 


Create one or more instances of the specified 
sync module type. 


CAUTION: Do not create multiple instances of 
the NTP sync module. 


selection_policy 


automatic| manual 
[initial-manual- 
instance] 


Use automatic or manual sync instance 
switching. 


Default: automatic. 


selection_holdoff_ 
interval 


<number> 


Specifies how many seconds to wait after 
detecting a better instance before selecting it. 
The holdoff interval prevents rapid switching 
between instances of very similar quality. 


Default: 10 seconds. 


selection_policy_rules 


<manual | state | no- 
alarms | user-priority | 
clock-class | total- 
accuracy | allan- 
variance | steps- 
removed>* 


Define the list of rules for the automatic 
selection policy 


message_log 


syslog |stderr| 


Specifies where to send messages generated by 


<filename> the application. 
Default: stderr. 
stats_log off | stdout | Specifies if and where to log statistics generated 
<filename> by the application. 
Default: disabled. 
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Table 2: Configuration options for sfptpd (continued) 


Description 


Output realtime information collected by the 
PTP remote monitor in JSON-lines format to this 
file (http://jsonlines.org). 


Default: disabled. 


json_stats 


<filename> 


Enable output of machine-readable stats in 
JSON-lines format. Same data as produced by 
the stats_log. 


daemon 


Run sfptpd as a daemon. 


Default: disabled. 


lock 


off|on 


Specifies whether to use a lock file to stop 
multiple simultaneous instances of the daemon. 


Default: enabled. 


sync_interval 


<number> 


Specifies the interval in 2<°U™"E"> seconds at 


which the clocks are synchronized to the local 
reference clock. 


local_sync_threshold 


<number> 


Specifies the threshold in nanoseconds of the 
offset between the system clock and a NIC clock 
over a 60s period to be considered in sync 
(converged). 


Default: 1us (hardware timestamping) 
Default: 100us (software timestamping) 


See also the instance-specific sync_threshold 
option for each of the sync modules. 
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Table 2: Configuration options for sfptpd (continued) 


Option Parameters 


clock_control slew-and-step | 
step-at-startup | 
no-step | no-adjust | 


step-forward 


Description 


Specifies how the clocks are controlled by sfptpd. 
Possible values are: 


e —slew-and-step: allow clock stepping as 
necessary 


e  step-at-startup: only allow the clock to 
be stepped at startup 


¢ no-step: never step the clock 


e no-adjust: do not make any adjustment to 
the clocks 


e step-forward: only step the clock forward. 
Default: slew-and-step. 


If astep mode is enabled, sfptpd will step a clock 
if the offset from the reference clock is greater 
than 1 second. 


Solarflare sfptpd is able to slew the clock at a 
maximum rate of 1 million ppb (1 part ina 1000) 
so slewing will take approximately 20 minutes to 
recover 1 second. The slew rate is not 
configurable. 


clock_list [<name> | 
<mac-address> | 


<clock-id>] 


Specifies the set of clocks that sfptpd should 

discipline. Specify by clock name or MAC address 

or clock ID: 

e the clock name can be shown using 
ethtool -T <interface> 

e the clock id is derived from the MAC address 


with OxFFFE in the middle, and is not so easy 
to use. 


Default: all clocks are disciplined. 


persistent_clock_ off|on 
correction 


Specifies whether to used saved clock frequency 
corrections when disciplining clocks. 


Default: enabled. 
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Option Parameters Description 
timestamping interfaces [<name>| Specifies the set of interfaces on which general 
<mac-address> | *] receive packet timestamping should be enabled. 

Specify by interface name or MAC address, or 
use * to enable receive timestamping on all 
interfaces that support it. 
Default: receive packet timestamping is disabled 
for all interfaces. 

timestamping_ disable _ off|on Specifies whether to disable timestamping on 

on_exit exit. This affects all interfaces specified with 
timestamping interfaces as well as the 
interface selected for PTP. 
Default: on. 

non_solarflare_nics off|on Enable PTP functionality and hardware 
timestamps of PTP packets from non-Solarflare 
adapter which must support PHC API. 
Default: off 

pid_filter_p <number> Secondary servo PID filter proportional term 
coefficient. 
Default: 0.4. 

pid_filter_i <number> Secondary servo PID filter integral term 
coefficient. 
Default: 0.03. 

trace_level <number> Specifies the trace level if the application has 


been built with trace enabled. 


Default: 0, meaning no trace. 
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5.4 Example configuration files 


This section describes some generic example configuration files: 


e Inthe tarball distribution, configuration files are located in the config 
subdirectory of the install directory. 


To run sfptpd using one of these configuration files: 


cd <install_dir> 
./sfptpd -i <interface> -f config/<filename>.cfg 


e Inthe RPM distribution, configuration files are located in the /usr/share/ 
doc/packages/sfptpd/config directory. 


To run sfptpd using one of these configuration files: 


cd /usr/share/doc/packages/sfptpd/ 
sfptpd -i <interface> -f config/<filename>.cfg 


default.cfg 


A general default configuration: 

e aPTP sync module is instantiated using the sync_module option 

e sfptpd is run as a daemon by setting the daemon option 

e the PTP instance is configured to act as a PTP slave using the ptp_mode option 


e any PTP traffic is transmitted using hybrid mode by using the 
ptp_network_mode option (see Hybrid mode on page 66). 
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Many instances configurations are supported on SFN7000 and later series adapters. 


Example using multiple types of sync module, plus multiple instances of the same 
type of sync module: 


three PTP sync modules are instantiated using the sync_module option: 


- each instance is allocated to a different PTP domain using the ptp_domain 


option 


- all instances are configured to act as a PTP slave by using the ptp_mode 
option in the generic [ptp] section 


- any PTP traffic is transmitted using hybrid mode by using the 
ptp_network_mode option in the generic [ptp] section (see Hybrid mode 
on page 66) 


a PPS sync module is instantiated using the sync_module option 


an NTP sync module is instantiated using the sync_module option: 


- the key-id and key-value are set for authenticating with NTP using the 
ntp_key option 


a Freerun sync module is instantiated using the sync_module option. 


5.5 Understanding the sfptpd startup sequence 


When the sfptpd daemon is started it will generate several lines of output. A typical 
startup sequence is shown below. The PTP slave that graduates from startup to a 
listening state and finally to a slave state. Line numbers have been added. 


[slave-server]# sfptpd -i enp1sefe -f config/sfptpd.conf 


1 


3 
4 


7 


2016-12-05 10:26:16.158651: info: Solarflare Enhanced PTP Daemon, 


version 3.0.0. 


1003 


2016-12-05 10:26:16.232403: info: no clock frequency correction file 
/var/lib/sfptpd/freq-correction-000f :53ff:fe@1:7ba4 


2016-12-05 10: 
2016-12-05 10: 


enp1se@f1) 


2016-12-05 10: 


enabled 


2016-12-05 10: 


timestamps 


2016-12-05 10: 


PTP_LISTENING 


8 
9 


2016-12-05 10: 
2016-12-05 10:26:18.837132: 


26:16.232669: 
26:16.235757: 


26:16.236050: 


26:16.236281: 


26:16.336741: 


26:16.337149: 


info: ptp ptp1: creating sync-instanc 


info: ptp: clock is phc@(enp1sefe/ 


info: interface enp1s@fe: SO_TIMESTAMPING 


info: using SO_TIMESTAMPING hardware 


notice: ptp ptp1: Now in state: 


info: selected sync instance ptp1 


info: new best master selected: 


00aQ : 69FF: Fe@c: 2eb5(unknown)/1 
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10 2016-12-05 10:26:18.837146: notice: now in state: PTP_SLAVE, 
best master: 00a0:69fF:fe@c:2eb5(unknown)/1 


11 2016-12-05 10:26:19.775070: info: received first Sync from Master 


12 2016-12-05 10:26:20.774850: notice: slewing the clock with the maximum 
frequency adjustment 


13 2016-12-05 10:26:20.774874: info: received first DelayResp from Master 


Each line is described in the following table. 


Table 3: sfptpd startup output 


Line # 


Description 


Version of sfptpd running. 


A frequency correction file holds the frequency correction value (PPB) 
that is currently being used to discipline a clock: 


¢ = On initial startup frequency correction files do not exist. 


e Once created by sfptpd for each clock, they are updated every 60 
seconds. 


e _ Files are preserved over server reboot and sfptpd restart. 


Create an instance of a PTP sync module named ptp1. 


The Local Reference Clock: 


e Onan X2 series, 8000 series or 7000 series adapter this identifies 
the PTP hardware clock in the form: phcO(ethX/ethY) where ethx 
is the active clock interface and ethY the second adapter clock 
interface on this adapter. Dual-port adapters share the same clock. 


5-6 


If hardware timestamps cannot be initialized sfptpd will revert to 
software timestamping using the system clock. 


The slave remains in a LISTENING state listening for Announce 
messages from any master clock on the network. The Best Master Clock 
algorithm is used to determine the most accurate master clock to which 
all slaves will synchronize. Any other master clock on the same network 
will go into passive mode and not send PTP messages once the best 
master has been selected. The Best Master Clock algorithm is 
implemented in such a way that all slaves will arrive at the same 
conclusion and select the same master clock. 


Before the adapter clock can begin synchronization with an upstream 
PTP master clock, the slave must receive an Announce message 
followed by Sync and Follow_up messages. 


The slave selects an initial sync module instance to use, as described in 
Sync module selection on page 19. 
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Table 3: sfptpd startup output (continued) 


Line # Description 


9-10 The slave moves to a SLAVE state once it has received an Announce 
message, and has selected the best master clock. 


The best master is identified by the following: 

e the clock identity (a UUID derived from the MAC address) 
e the clock host 

e a forward slash ‘/’ 


e — the port number (in hex) of the upstream master (usually 1, 
because most Grandmasters have only one PTP port). 


11-13 Theslave accepts PTP Sync messages from the master and starts to 
synchronize the adapter clock(s) with the master clock. 


As the slave begins to synchronize the adapter clock with the external 
master clock, the offset (master>slave), one-way-delay 

(slave master), and frequency correction values are output. See 
Understanding sfptpd output below. 


5.6 Understanding sfptpd output 
Once it has reached a SLAVE state, sfptpd generates output describing the current 
state of the synchronization. 


By default sfptpd will output the offset data to stdout. The user can redirect 
stats_log output to file using the configuration file stats_log parameter. 


When logging the stats_1log to file, it is possible to override this using the -v 
option on the sfptpd command line to cause stats to display on stdout. 
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The following examples are from a PTP sync module on a slave server. The 
timestamp at the start of the line has been omitted. Other sync modules produce 
similar output. 
[ptp1:gm->phc@(enp1se@fe)], offset: 0.500, freq-adj: -663.353, in-sync: 1, 
one-way-delay: 7534.500, grandmaster-id: 00a0:69ff:fe@c:2eb5 
[phc@(enp1s@fO/enp1s@f1)->system], offset: 2.812, freq-adj: 46378.715, 
in-sync: 1 
Table 4: sfptpd offset output 
Parameter Description 
[ptp1:gm-> A line of output generated for measurements 
phc@(enp1sefé@) ] between the external master clock and the Local 
Reference Clock: 
e the -> separator indicates that the right-hand 
clock is being synchronized to the left-hand one 
¢ a-- separator indicates no synchronization 
between clocks 
offset The current offset (nanoseconds) between the 
master clock and the slave clock identified at the 
start of the line. 
Immediately following startup this is expected to be 
a large value, but will gradually decrease until it 
settles to its lowest value. Synchronization can 
typically take between 15-30 minutes. 
freq-adj The current rate (PPB) at which the clock is being 
disciplined by sfptpd. This value is stored in the 
freq-correction file for this clock every 60 seconds. 
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Table 4: sfptpd offset output (continued) 


Parameter 


in-sync 


Description 


The in-sync flag will be 1 when the offset between 
master and slave clocks is below the value of the 
local_sync_threshold option (default (hardware 
timestamping 1s) for a period of 1 minute. 


The local_sync_threshold when using software 
timestamping is 100us. 
The in-sync flag will be 0 before the above condition 


is true. 


The in-sync flag will be 0 if the offset becomes 
greater than the local_sync_threshold. 


The in-sync flag will change to 0 if an alarm condition 
exists on the server to indicate problems in the PTP 
network e.g. PTP messages not being sent or 
received by the slave server. 


Check the topology file for current alarms status. 


one-way-delay 


The current one-way-delay (nanoseconds) between 
master and slave servers. 


This value should not be zero, but, once the server is 
synchronized, it should remain fairly stable: 


e If the value is zero, check that Delay_Req and 
Delay_Resp message are being sent and 
received. 


e If the value does not change at all over an 
extended period, check the Delay_Req interval. 


Check the topology file for current alarms status. 


parent-id UUID of the upstream master clock. This may be a 
boundary clock or the GM Master clock. 

gm-id Master clock UUID derived from its MAC address. 

[phc@(enp1sefe/ A line of output generated for measurements 


enp1s@f1)->system ] 


between the Local Reference Clock and the server 
system clock: 


e —_ the -> separator indicates that the right-hand 
clock is being synchronized to the left-hand one 


e a-- separator indicates no synchronization 
between clocks 
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performance data is accumulated in files created in the following directory: 


/var/lib/sfptpd 


From these files the user is able to monitor the performance and status of sfptpd 


and the PTP server. 


Table 5 lists statistics files created by sfptpd. 


Table 5: Statistics files created by sfptpd 


File name 


freq-correction-system 


Persistent 


YES 


Description 


Contains the frequency 
correction value used to 
discipline the server system 
clock. 


See 


page 46 


freq-correction-<UUID> 


YES 


Contains the frequency 
correction value used to 
discipline the clock identified 
by the UUID identifier. 


page 46 


interfaces 


NO 


Identifies all adapters, and 
their timestamping 
capabilities. 


page 46 


ptp-nodes 


NO 


Identifies all PTP clocks 
present in the network. 


page 46 


state-system 


NO 


The contents of this file 
depend on the current PTP 
mode. In slave mode this file 
contains historical offset and 
status data for the slave 
server. 


page 47 


state-<sync_module> 
or 
state-<clock_UUID> 


NO 


Contains status information 
relevant to the identified 
sync module or clock. 


page 47 


stats-system 


NO 


Contains accumulated PTP 
performance data for the 
server including counts of the 
PTP message types sent and 
received. 


page 48 
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Table 5: Statistics files created by sfptpd (continued) 


File name Persistent Description See 
stats-<sync_module> NO Contains aggregated/ page 48 
or summarized synchronization 
stats-<clock_UUID> data for the identified sync 


module or clock. 


From version 3.2.1 stats files 
are automatically generated 
and in txt and JSON formats. 


topology NO PTP clock hierarchy diagram page 45 
showing all clocks local to the 
slave server and PTP network 
elements between the slave 
and master clock. 


Identifies the timestamping 
mode (HW or SW). 


Displays active PTP alarms. 


version NO Contains the sfptpd version. page 49 


File Persistence: 


Of all the files created, the freq-correction-* files are persistent and will be 
preserved over sfptpd restart and over server reboot. All other files are non- 
persistent and are created when sfptopd is started. 


File Update Rate: 


The topology file and ptp-nodes file will update immediately to reflect changes in 
the ptp network topology or alarm conditions. 


Alarms will also be updated immediately in clock state files. 


Stats files are updated every 60 seconds. 
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Topology file 


When viewed on a sfptpd slave server, the topology file presents a PTP clock 
hierarchy diagram showing all clocks local to the slave server and PTP network 
elements between the slave and master clock. 


e A PTP Boundary Clock would be visible in the file as a parent to the slave. 
e APTP Transparent Clock will also be visible in the topology file. 


The topology file identifies the current state of the selected slave clock, the interface 
being used to receive PTP messages and the timestamping mode being used. Each 
clock in the topology is identified by its UUID which is derived from its MAC address. 
Nanosecond values between clocks are the offset values recorded during the last file 
update. 


The following output is a example of a topology file currently showing alarm states. 


state: ptp-slave-alarm 

alarms: no-sync-pkts no-delay-resps 
interface: eth2 (eth2) 
timestamping: hw 


grandmaster 
Q00Ff : 53FF:fe16:0474/1 
| 
| 


-170.000 ns 


Vv 
phc@(eth2/eth3) 
Q00F :53FF:Fe21:9bbe 


-52.688 ns 


Vv 
system 
system 


In the above example, the slave is in an alarm state indicating that Sync messages 
and Delay_Response messages are not currently being received from the master 
clock. 


The topology file can be periodically monitored, for example, using a script to 
extract key fields, to monitor the current connection state and synchronization 
status of the PTP slave. 


G) NOTE: During normal operation the topology file is updated every second. 
However, alarm or state changes are reflected in the file immediately. 
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PTP nodes file 


The ptp-nodes file identifies all PTP clocks present in the network. 


# cat ptp-nodes 
| state | clock-id | port-number | domain | 


Interfaces file 


The interfaces file identifies all adapters in the PTP slave server and identifies the 
timestamping capabilities. In the following example, Solarflare adapter interfaces 
are identified as being hardware timestamping capable. 


# cat interfaces 


interface | ptp-caps | pkt-timestamping-caps product-name 
etho - - Broadcom NetXtreme II Ethernet Controller 
eth1 - - Broadcom NetXtreme II Ethernet Controller 
eth2 hw - Solarflare SFN5322F SFP+ Time Stamping Server Adapter 
eth3 - - Solarflare SFN5322F SFP+ Time Stamping Server Adapter 
eth4 hw hw Solarflare SFN7122F SFP+ Server Adapter 
eth hw hw Solarflare SFN7122F SFP+ Server Adapter 


Frequency correction files 


Each freq-correction file contains the frequency correction values used to 
discipline a clock. The following table describes the values in the file: 


Table 6: File: freq-correction-<UUID> 


Name Description 


freq-correction-<UUID> This identifier is constructed from the hardware 
address of the clock port. 


value This is the frequency correction value used to 
discipline the clock. The value is updated once per 
minute when the clock is in sync with its 
synchronization time source. 


This file persists over server reboot and sfptpd 
restart. Following either event, the frequency 
correction value is used to recommence 
disciplining of the clock ensuring faster re- 
convergence. 
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State files 
Each state file contains status information. 

G) NOTE: The contents of this file depend on the current PTP mode and position of the 
clock in the PTP hierarchy. A state file exists for each sync module and for the server 
system clock. 

The following table describes the values in a typical state file: 
Table 7: File: state-<sync_module> or state-<clock_UUID> 

Name Description 

clock-name identify the clock using this clock-id. 

clock-id the clock UUID. 

state ptp_slave | ptp_master 
also identifies if the clock is subject to any current 
alarms. 

interface identifies the PTP clock interface. 

timestamping current timestamping mode. 

offset-from -master Offset (nanoseconds) of LRC from the master 
clock. 

one-way-delay One-way-delay (nanoseconds) between LRC and 
master clock. 

freq-adjustment-ppb Current frequency correction value used to 
discipline this clock. 

observed-drift Drift in nanoseconds of slave LRC to master clock. 

in-sync 0: observed offset > local_sync_threshold option 
1: observed offset < local_sync_threshold option 
(The default for the local_sync_threshold is 1 us. 
when using hardware timestamping. 100us for 
software timestamping) 

steps-removed steps removed from the master clock. 

parent-clock-id UUID of the PTP parent clock. If there is a 
boundary clock between LRC and the master clock 
this will identify the boundary clock. 

parent-port-num Port number relayed to the server by the master 
clock in the SourcePortID parameter. 

grandmaster-id The UUID grandmaster clock constructed from the 
grandmaster hardware address. 
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Table 7: File: state-<sync_module> or state-<clock_UUID> (continued) 
Name Description 
grandmaster-clock-class The current Grandmaster class value. 
grandmaster-clock-accuracy The current Grandmaster accuracy value. 
grandmaster-priority1 The current Grandmaster priority 1 value. 
grandmaster-priority2 The current Grandmaster priority 2 value. 
current-utc-offset The current UTC offset in seconds from TAI value 
specified in the config file with the ptp-utc-offset 
value. 
leap-59 1 indicates a leap second is scheduled. 
leap-61 1 indicates a leap second is scheduled. 
Stats files 


@ 


Each stats file contains accumulated data and statistics. 


NOTE: The contents of this file depend on the current PTP mode and position of the 
clock in the PTP hierarchy. A stats file exists for each sync module and also for the 


server system clock. 


The following table describes the values in a typical stats file: 


Table 8: File: stats-<sync_module> or stats-<clock_UUID> 


Name 


offset-from-master (ns) 


one-way-delay (ns) 


freq-adjustment-ppb (ppb) 


Description 


Mean, min and max values are accumulated for 
these statistics. The number of samples taken 
during each period is recorded as is the start/end 
time of each sample period. 
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Table 8: File: stats-<sync_module> or stats-<clock_UUID> (continued) 


Name Description 

synchronized The stats file retains accumulated counts of each 
k F PTP message type sent or received from the 

announce-pkts-txe Sania 

announce-pkts-rxed The statistics file also records the number of PTP 


packets sent of received without being hardware 
timestamped. 


announce-timeouts 


ee aay The number of samples taken during each period 


sync-pkts-rxed is recorded, as is the start/end time of each 
sample period. 


sync-pkt-timeouts 


follow-up-pkts-txed 


follow-up-pkts-rxed 


follow-up-timeouts 


delay-req-pkts-txed 


delay-req-pkts-rxed 


delay-resp-pkts-txed 


delay-resp-pkts-rxed 


delay-resp-timeouts 


delay-mode-mismatch- 
errors 


clock-steps 


tx-pkt-no-timestamp 


rx-pkt-no-timestamp 


crx-pkt-no-timestamp Mean, minimum, maximum and standard 
deviation values are accumulated for these 
packets. The number of samples taken during 
each period is recorded, as is the start/end time of 
each sample period. 


Version file 


The version file contains the sfptpd version. 
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The section explains and demonstrates the various data logging options available 
with sfptpd. 


Event logging 


PTP events including startup events can be directed to the syslog or stderr by 
enabling the following option in the configuration file: 


message log [syslog | stderr] 


Stats logging 


The following option is used to enable stats logging and display output on stdout or 
redirect output to a file: 


stats_log [off | stdout | filename] 


For more information, see Understanding sfptpd output on page 40. 


PTP packet capture 


Enabling the following option in the configuration file will display the contents of 

PTP packets sent and received by the sfptpd process: 

ptp_pkt_dump 
)\ CAUTION: This option produces extensive output for each received PTP packet, as 


the following example of a PTP SYNC message demonstrates, and should only be 
used for debugging purposes. 


2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 
2012-12-20 


18 
18 
18 
18 
18 
18 
18 
18 
18 
18 
18 
18 


18: 


145: 
145: 
145: 
145: 
145: 
145: 
145: 
145: 
145: 
145: 
145: 
145: 
45: 


29 
29 
29 
29 
29 
29 
29 
29 
29 
29 
29 
29 


-496035 
-496035 
-496035 
-496035 
-496035 
-496035 
-496035 
-496035 
-496035 
-496035 
-496035 
-496035 
29. 


496035 


m 
m 
m 
m 
m 
m 
m 
m 
m 
m 
m 
m 
m 


sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 
sgDebugHeader: 


messageType @ 

versionPTP 2 

messageLength 44 

domainNumber @ 

flags @2 00 

correctionfield @ 
sourcePortIdentity.clockIdentity Q00f:53ff:fe16:0474 
sourcePortIdentity.portNumber 1 
sequenceld 94 

controlField @ 
logMessageInterval @ 


sgDebugSync: originTimestamp.seconds 1356029129 
sgDebugSync: originTimestamp.nanoseconds 856792000 


G) NOTE: This is different to using tcpdump which will capture packets received/sent 
at an interface. 
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Trace level 


The following option is used to specify the trace level if the application has been 
built with trace enabled: 


trace_level <number> 


By default this is 0, meaning no trace. Setting a higher value enables trace. 


PTP trace level 
The following option is used to specify the PTP trace level: 
ptp_trace <number> 


By default this is 0, corresponding to off. Setting a higher value makes trace more 
verbose, up to a maximum value of 3. 


5.9 Remote Monitoring 


sfptpd from version 3.2 supports extensive local and remote monitoring 


capabilities, producing human-readable and machine-readable statistical/state/ 
alarm data. 


For details of the monitoring options refer to Chapter 16 on page 117. 
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6 PTP Sync Module 


6.1 Description 


Using the PTP sync module, sfptpd is receiving and sending PTP messages to an 
upstream PTP master clock. 


6.2 Sync characteristics 


This sync module can be configured for different roles: 
e —_ PTP slave, with optional fallback 

e — PTP master, with optional fallback 

e — PTP boundary clock 

e PTP multiple masters. 


Examples of each are provided (see Example configuration files on page 62). 


Loss of PTP link network connection 


If the network connection to the external master clock is lost at any point, an alarm 
is triggered in the corresponding PTP sync module. It can no longer be selected as 
the time source, unless all other sync modules are in the alarm state (e.g. at startup): 


e —_ If other sync modules are instantiated and functioning correctly, sfptpd will 
select one of them as the time source. 


e If no other sync module is available, Solarflare’s sfptpd continues to discipline 
all clocks in the system including the system clock. 


sfptpd maintains a clock frequency correction file for each clock in a server: 


- If the LRC is a Solarflare PTP adapter, sfptpd will continue to discipline the 
LRC using the LRC frequency correction file. sfptpd will ensure that other 
clocks, hardware or system, will continue to synchronize to the LRC. 


- | When there is no Solarflare PTP adapter in the server, or if the LRC is the 
system clock, sfptpd will continue to discipline the system clock using the 
system clock frequency correction file. 


Following restoration of the network link, the PTP sync modules receives PTP 
packets and the alarm is no longer triggered. It is eligible for selection as a time 
source, and sfptpd will resume normal discipline procedure. 
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6.3 PTP over VLAN 


Solarflare Enhanced PTP supports PTP packets over tagged 802.1Q Virtual Local 
Area Network (VLAN) interfaces. Users should consult the relevant OS 
documentation for VLAN configuration instructions. Assuming interface eth2.12@ is 
a network interface configured with VLAN tag 120, the following example identifies 
sfptpd VLAN configuration. 


./sfptpd -i eth2.120 -f config/ptp_slave.cfg 


6.4 PTP over bonded interfaces 


Solarflare adapters and sfptpd support PTP packets over bonded interfaces in an 
active/standby mode. In addition, sfptpd also supports bonding over LACP 
(802.3ad) bonding. 


Bonding of Solarflare interfaces employs the Linux bonding driver. Multiple ports 
can be included into a single bond where one port is selected as the active interface 
and all others are standby. 


e — sfptpd will detect which port is active and which ports are passive in the bond. 


e sf ptpd will discipline the high precision clock on the active port’s network 
adapter. 


e —sfptpd will discipline the clocks of passive ports from the active adapter’s 
clock. 


e Via the bonding driver the user can select the active port (and therefore clock). 


e Abond can include non-PTP capable Solarflare ports. 


sfptpd will switch to software time-stamping when a non-hardware time- 
stamping port becomes active. 


e Abond can include non Solarflare ports. 


sfptpd will switch to software time-stamping when a non-Solarflare port 
becomes active. 


e  Abond can include any number of ports. 


G) NOTE: There are limitations to combining PTP and non-PTP ports in the same bond. 
See Bonding on page 128 for details. 
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Bonding configuration 


Bonding of Solarflare interfaces is handled by the standard Linux bonding driver. 
Users should refer to http://www.kernel.org/doc/Documentation/networking/ 
bonding.txt for details of alternative methods for bonding configuration. The 
following example is a manually bonding configuration using ifenslave: 


# modprobe bonding miimon=100 mode=active-backup primary=eth5 
# ifconfig bond@ 172.16.136.27/21 
# ifenslave bond@ eth@ eth1 


To run sfptpd over the bonded interfaces: 


./sfptpd -i bond<N> -f config/ptp_slave.cfg 


Action on active port failover 


The active port in the bonding interface identified on the command line with the -i 
option is the active clock. In the event of failure of the active port: 


e If the standby port is a PTP capable port, synchronization will continue. The 
standby port clock is synchronized to the active port clock. 


e If the standby port is a port that does not support hardware timestamps, the 
system clock becomes the LRC and sfptpd uses software timestamping. 


Link States, Interface Hotswap 


Uses the Linux Netlink interface to improve robustness and availability of sfptpd 
when used over Linux bonded interfaces. 


e = As long as there is a link available, sfptpd will remain active in the PTP_SLAVE 
state. 


e If all bonded links are down sfptpd will remain active in the PTP_LISTENING 
state and will activate the no-interface alarm. 


e When a failed bond link is restored, sfptpd will resume, changing to a 
PTP_SLAVE state if the previous state was PTP_LISTENING. 


e If an adapter/interface is swapped out/in, sfptpd will identify the new 
interface(s) and continue. 
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6.5 Configuration options 


The PTP sync module has generic options that apply to all instances of the module, 
and further instance-specific options that apply only to a single instance of the 
module. 


Generic configuration options 


Table 9 shows the generic configuration options for the PTP sync module. These 
must be in a section labeled [ptp]. 


Table 9: Generic configuration options for the PTP sync module 


Option Parameters Description 


ptp_mgmt_msgs disabled | read-only Configures PTP Management Message support. 
Possible values are: 


e disabled: management messages disabled 


e read-only: only requests to read 
information (GET) will be accepted. 


See PTP management messages on page 67. 


By default this is disabled. 


ptp_max_foreign_records <number> The maximum number of PTP foreign master 
records a node is able to store simultaneously. 


The default value is 16. 


ptp_uuid_filtering off|on Specifies whether to use hardware packet 
filtering against parent UUID for the SFN6322F 
adapter. 


By default this is enabled. 


ptp_domain_filtering off|on Specifies whether to use hardware packet 
filtering against PTP domain for the SFN6322F 
adapter. 


The default value is on. 


state_path directory Directory in which to store sfptpd state data. 
Defaults to /var/lib/sfptpd. 


Issue 8 © Copyright 2020 Xilinx, Inc 55 


, Solarflare Enhanced PTP User Guide 
SOLARFLARE® 
A XILINX COMPANY PTP Sync Module 


Instance-specific configuration options 


Table 10 shows the instance-specific configuration options for the PTP sync module. 
These must be in a section labeled with the instance name. 


Table 10: Instance-specific configuration options for the PTP sync module 


Option Parameters Description 


ptp_mode slave | master Specifies the PTP mode of operation. 


The default mode is slave. 


interface <interface-name> Specifies the name of the interface that PTP 
should use. 
priority <number> Relative priority of sync module instance.* 


Smaller values have higher priority. 


The default is 128. 


sync_threshold <number> Threshold in nanoseconds of the offset from 
the clock source over a 60s period to be 
considered in sync (converged). 


The default is 1000ns with hardware 
timestamping and 100000ns with software 
timestamping. 


See also the local_sync_threshold option in 
Table 2 on page 33. 


ptp_pkt_dump — Dump each received PTP packet in detail. 

ptp_pps_log — Enable logging of PPS measurements. 

ptp_tx_latency <number> Specifies the outbound latency in 
nanoseconds. 


Use this option and the ptp_rx_latency option 
to correct for any network asymmetry. See 
Chapter 15 on page 111. 


ptp_rx_latency <number> Specifies the inbound latency in nanoseconds. 


Use this option and the ptp_tx_latency option 
to correct for any network asymmetry. See 
Chapter 15 on page 111. 


ptp_delay_mechanism end-to-end | Peer delay mode. 


peer-to-peer The default mode is end-to-end. 


ptp_network_mode multicast | hybrid | Network mode. See Network transmission 
hybrid-no-fallback. modes on page 65. 


The default mode is hybrid. 
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Table 10: Instance-specific configuration options for the PTP sync module (continued) 


Option Parameters Description 
ptp_ttl <number> The TTL value to use in transmitted PTP 
packets. 


The default value is 64. 


ptp_utc_valid_handling default | Controls how the UTC offset valid flag is used. 
ignore | The specification is ambiguous in its 
prefer | description of the meaning of the UTC offset 


valid flag, and this has resulted in varying 
different implementations. In most 

override implementations, if the UTC offset valid flag is 
not set then the UTC offset is not used, but in 
others the UTC offset valid is an indication that 
the master is completely confident that the 
UTC offset is correct. Various options are 
supported: 


require | 


e =>. de fau 1t: if UTCV is set use the UTC offset, 
otherwise do not use it 


e ignore: donot used the UTCV flag, always 
apply the indicated UTC offset 


e prefer: prefer grand masters that have 
the UTCV flag set above those that don't 


e require: do not accept GMs that do not 
set UTCV. 


e override <seconds>: set a fixed value for 
the UTC offset value. This option will 
ignore any UTC offset value from the 
upstream PTP master clock. 


CAUTION: The user must ensure the override 

A value is set to the current UTC offset. The UTC 
offset value increases by 1 second on the 
occasion of each leap second. 


See UTC offset on page 68. 


ptp_domain <number> Specifies the PTP domain. 


The default value is 0. 
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Table 10: Instance-specific configuration options for the PTP sync module (continued) 


Option 


ptp_timing acl_allow 


Parameters 


<ip-address-list> 


Description 
Access control permit list for timing packets: 


The format is a space- or comma-separated list 
of network prefixes in a.b.c.d/x notation, 
where a.b.c.d is the subnet and x is the mask. 
For single IP addresses, 32 should be specified 
for the mask. 


See Access control lists on page 69. 


ptp_timing acl_deny 


<ip-address-list> 


Access control deny list for timing packets: 


The format is a space- or comma-separated list 
of network prefixes in a.b.c.d/x notation, 
where a.b.c.d is the subnet and x is the mask. 
For single IP addresses, 32 should be specified 
for the mask. 


See Access control lists on page 69. 


ptp_timing acl_order 


permit-deny| 
deny-permit 


Access control list evaluation order for timing 
packets. 


See Access control lists on page 69. 


ptp_mgmt_acl_allow 


<ip-address-list> 


Access control permit list for management 
packets: 


The format is a space- or comma-separated list 
of network prefixes in a.b.c.d/x notation, 
where a.b.c.d is the subnet and x is the mask. 
For single IP addresses, 32 should be specified 
for the mask. 


See Access control lists on page 69. 


ptp_mgmt_acl_deny 


<ip-address-list> 


Access control deny list for management 
packets: 


The format is a space- or comma-separated list 
of network prefixes in a.b.c.d/x notation, 
where a.b.c.d is the subnet and x is the mask. 
For single IP addresses, 32 should be specified 
for the mask. 


See Access control lists on page 69. 


ptp_mgmt_acl_order 


permit-deny| 
deny-permit 


Access control list evaluation order for 
management packets. 


See Access control lists on page 69. 
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Table 10: Instance-specific configuration options for the PTP sync module (continued) 


Option Parameters Description 
ptp_announce_interval <number> The PTP Announce packet interval in 2<number> 
seconds. 


The default value is 1. 


ptp_announce_timeout <number> The PTP Announce packet receipt timeout as a 
number of Announce packet intervals. 


The default value is 6. 


ptp_sync_pkt_interval <number> The PTP Sync packet interval in 24<number> 
seconds. 


The default value is 0 (1 second). 


ptp_sync_pkt_timeout <number> The PTP Sync packet receipt timeout as a 
number of Sync packet intervals. 


The default value is 6. 


ptp_delayreq_interval <number> The PTP Delay Request / Peer Delay Request 
packet interval in 2<PU™e"> seconds. 


If specified for a PTP slave, this overrides the 
value communicated to the slave from the 
master. 


ptp_delayresp_ timeout <number> The PTP Delay Response receipt timeout in 
2<number> seconds 


The default value is -2 (250ms). 


ptp_bmc_priority1 <number> Best Master Clock algorithm, priority1 
parameter. Used when in PTP master mode. 


ptp_bmc_priority2 <number> Best Master Clock algorithm, priority2 
parameter. Used when in PTP master mode. 


ptp_trace <number> PTP trace level: 
¢  Ocorresponds to off 
e 3corresponds to maximum verbosity. 


The default value is 0. 
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Table 10: Instance-specific configuration options for the PTP sync module (continued) 


Option 


pid_filter_p 


Parameters 


<number> 


Description 


Secondary servo PID filter proportional term 
coefficient. 


The default value is 0.2. 


pid_filter_i 


<number> 


Secondary servo PID filter integral term 
coefficient. 


The default value is 0.003. 


remote_monitor 


Enable the remote monitor. Collects Slave 
Event Monitoring messages 


json_remote_monitor 


<filename> 


Output realtime information collected by the 
PTP remote monitor in JSON-lines format. 


Default: disabled. 


mon_monitor_address 


<address> 


Address of monitoring station to which to send 
signaling messages with slave event monitoring 
data for the mon_rx_sync_timing_data, 
mon_rx_sync_computed_data and 
mon_tx_event_timestamps commands. 


Default is multicast. 


mon_rx_sync_timing data 


none 


sfptpd will generate/output the timestamps for 
received Sync messages. 


mon_rx_sync_computed_d 
ata 


none 


sfptpd will generate/output the computed 
offset from the master clock. 


mon_tx_event_timestamps 


none 


sfptpd will generate/output the timestamp for 
transmitted Delay Request messages. 


outlier_filter_size 


<number> 


Number of data samples stored in the offset 
from master filter. The valid range is [5, 60]. 


The default value is 60. 


outlier_filter_adaption 


<number> 


Controls how outliers are fed into the offset 
from master filter. The valid range is [0, 1]: 


e —avalue of 0 means that outliers are not fed 
into the filter (not recommended) 


e —avalue of 1 means that each outlier is fed 
into the filter unchanged 


e values between result in a portion of the 
value being fed in. 


The default value is 1. 
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Option 


Table 10: Instance-specific configuration options for the PTP sync module (continued) 


Parameters Description 


mpd_filter_size <number> Number of data samples stored in the mean 


path delay filter. The valid range is [1, 25]: 
e ~—avalue of 1 means that the filter is off 


e higher values will reduce the adaptability 
of PTP but increase its stability. 


The default value is 8. 


mpd_filter_ageing <number> Controls the ageing of samples in the mean 


path delay filter. The ageing is expressed in 
units of nanoseconds per second. 


The default value is 2.0ns/s, 


fir_filter_size <number> Number of data samples stored in the FIR filter. 


The valid range is [1, 128]. 
e ~—avalue of 1 means that the filter is off 


e higher values will reduce the adaptability 
of PTP but increase its stability. 


The default value is 1, and so the filter is off. 


This is the user priority for this sync instance within this daemon and is unrelated to the IEEE- 
1588 specification PTP 'priority1' and 'priority2' values. 


If the priority is specified in a generic section - outside of a specific (named) sync instance, the 
priority will apply to all sync modules of the same type e.g. 


sync_module ptp namedptp1 namedptp2 


[ptp] 
// this applies to all PTP type sync module instances 
priority 128 


[namedptp1] 
// this applies only to this instance - and takes priority over any generic value 
priority 127 


[namedptp2] 
// this applies only to this instance - and takes priority over any generic value 
priority 126 
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6.6 Example configuration files 


This section describes the example configuration files that use PTP as the only sync 
module, or as the principal one. See Example configuration files on page 37. 


G) NOTE: For descriptions of the remaining files in the config directory, see the 
chapters describing the other sync modules. 


ptp_slave.cfg 
This file shows how to configure sfptpd to act as a PTP slave: 
e aPTP sync module is instantiated using the sync_module option 


e the PTP sync module is configured to act as a PTP slave by using the ptp_mode 
option 


e _ the slave also has ptp_tx_latency and ptp_rx_latency options that can be 
used to compensate for latency through the interface (see Chapter 15 on 
page 111) 


e any PTP traffic is transmitted using hybrid mode by using the 
ptp_network_mode option (see Hybrid mode on page 66). 


ptp_slave_ntp_fallback.cfg 


This file shows how to configure sfptpd to act as a PTP slave, with fallback to NTP: 


e anNTP sync module is instantiated using the sync_module option 


- the key-id and key-value are set for authenticating with NTP using the 
ntp_key option 


e aPTP sync module is instantiated using the sync_module option 


e the PTP sync module is configured to act as a PTP slave by using the ptp_mode 
option 


e _ the slave also has ptp_tx_latency and ptp_rx_latency options that can be 
used to compensate for latency through the interface (see Chapter 15 on 
page 111) 


e any PTP traffic is transmitted using hybrid mode by using the 
ptp_network_mode option (see Hybrid mode on page 66). 


Failover to NTP 


sfptpd will failover from PTP to NTP if there are active PTP alarms: 


pps-no-signal 
pps-seq-num-error 
no-time-of-day 
pps-bad-signal 
no-sync-pkts 
no-follow-ups 
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no-delay-resps 
no-pdelay-resps 
no-pdelay-resp-follow-ups 
no-tx-timestamps 
no-rx-timestamps 
no-interface 
clock-ctr1-failure 


See PTP Alarms on page 141 for a description of alarms. 
If alarm conditions clear, sfptpd will resume synchronization using PTP. 


The selection_holdoff_interval (seconds) can be set to prevent immediate failover 
caused when only one or two PTP packets are not received. 


sfptpd will NOT failover to NTP because of synchronization accuracy. 


ptp_slave_multiple.cfg 


Many instances configurations are supported on SFN7000 and later series adapters. 


This file shows how to configure sfptpd to act as a PTP slave listening to multiple 
PTP domains: 


e three PTP sync modules are instantiated using the sync_module option: 

e — each instance is allocated to a different PTP domain using the ptp_domain 
option 

e all instances are configured to act as a PTP slave by using the ptp_mode option 


in the generic [ptp] section 


e —allinstances use ptp_tx_latency and ptp_rx_latency options in the generic 
[ptp] section that can be used to compensate for latency through the interface 
(see Chapter 15 on page 111) 


e any PTP traffic is transmitted using hybrid mode by using the 
ptp_network_mode option in the generic [ptp] section (see Hybrid mode on 
page 66). 


ptp_master_ntp.cfg 


This file shows how to configure sfptpd to act as a PTP master, using the NTP 
daemon as the reference: 


¢ anNTP sync module is instantiated using the sync_module option 


- the key-id and key-value are set for authenticating with NTP using the 
ntp_key option 


¢ aPTP sync module is instantiated using the sync_module option 


e the PTP sync module is configured to act as a PTP master by using the ptp_mode 
option 


Issue 8 


© Copyright 2020 Xilinx, Inc 63 


SOLARFLARE® 


A XILINX COMPANY 


Solarflare Enhanced PTP User Guide 
PTP Sync Module 


the master also has ptp_tx_latency and ptp_rx_latency options that can be 
used to compensate for latency between the NTP daemon and interface (see 
Chapter 15 on page 111) 


any PTP traffic is transmitted using hybrid mode by using the 
ptp_network_mode option in the generic [ptp] section (see Hybrid mode on 
page 66). 


G) NOTE: NTP uses UTC time - not atomic (TAI) time. When sfptpd is configured as a 
master clock, it uses UTC time. Therefore ensure that the ptp_utc_offset parameter 
in the sfptpd master config file is set to a value of 0. 


ptp_master_freerun.cfg 


This file shows how to configure sfptpd to act as a PTP master, using the NIC clock 
as the reference: 


a Freerun sync module is instantiated using the sync_module option 
a PTP sync module is instantiated using the sync_module option 


the PTP sync module is configured to act as a PTP master by using the ptp_mode 
option 


the master also has ptp_tx_latency and ptp_rx_latency options that can be 
used to compensate for latency between the clock and interface (see Chapter 
15 on page 111) 


ptp_boundary.cfg 


This file shows how to configure sfptpd to act as a PTP boundary clock: 


two PTP sync modules are instantiated using the sync_module option 


one instance is associated with a particular interface, and is configured to act 
as a PTP master by using the ptp_mode option 


the other instance is associated with a different interface, and is configured 
to act as a PTP slave by using the ptp_mode option 


the slave also has ptp_tx_latency and ptp_rx_latency options that can be 
used to compensate for latency between the two interfaces (see Chapter 15 on 
page 111) 

both instances are allocated to the same PTP domain by using the ptp_domain 
option in the generic [ptp] section 


any PTP traffic is transmitted using hybrid mode by using the 
ptp_network_mode option in the generic [ptp] section (see Hybrid mode on 
page 66) 


NOTE: Using the ptp_boundary.cfg, sfptpd will operate as both master and slave 

G) simultaneously within the same or different PTP domains. This boundary clock 
mode is not intended to meet all the requirements of a boundary clock as specified 
in the IEEE-1588 specification. 
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G) NOTE: When acting as a master clock, sfptpd uses UTC time and not TAI time. 


ptp_domain_bridge.cfg 
This file shows how to configure sfptpd to act as a PTP domain bridge: 
¢ two PTP sync modules are instantiated using the sync_module option 
¢ one instance is configured to act as a PTP master by using the ptp_mode option 


e the other instance is configured to act as a PTP slave by using the ptp_mode 
option 

e — each instance is allocated to a different PTP domain using the ptp_domain 
option 


e _ the slave also has ptp_tx_latency and ptp_rx_latency options that can be 
used to compensate for latency between the two interfaces (see Chapter 15 on 
page 111) 


e any PTP traffic is transmitted using hybrid mode by using the 
ptp_network_mode option in the generic [ptp] section (see Hybrid mode on 
page 66). 


6.7 Configuration options in detail 


Network transmission modes 
The network transmission mode for an instance of the PTP sync module is set by the 
ptp_network_mode option in the configuration file. It can have the following values: 
e multicast 
e = hybrid. 
e — hybrid-no-fallback 


Multicast mode 


The is the standard PTP mode whereby all PTP packets between master and slave 
are sent as multicast. The implication is that all slaves receive all Delay_Req, 
Delay_Resp message pairs between all other slaves and the PTP master clock 
thereby increasing the amount of traffic on the network. The traffic level generated 
by multicast transmission is usually not a problem on smaller networks employing 
only a few PTP slaves. 


Multicast mode can be selected for sfptpd using the configuration file option 
ptp_network_mode multicast. 
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Hybrid mode 


PTP hybrid mode allows a PTP slave clock to use unicast transmission to send 
Delay_Req messages to the master clock which, in turn, will respond with a unicast 
Delay_Resp packet direct to the relevant slave. This reduces the level of PTP traffic 
on the network which can be a factor when scaling to larger networks employing 
many PTP slaves. Hybrid mode only requires the network to support multicast 
transmission from master to slave. 


A sfptpd slave, using hybrid mode, will make three attempts to contact a master 
clock when sending a Delay_Req message using unicast transmission. If the master 
clock fails to respond to unicast transmissions the sfptpd slave will revert to 
multicast transmission. If hybrid mode communication is possible with the master 
clock, the slave will remain in hybrid mode until/if a new master clock is selected. 


sfptpd will generate an error message if the PTP master fails to respond to the 
unicast Delay_Req message. Error messages will go to stderr or to syslog, 
depending on the logging configuration. 


Hybrid mode is the default mode for sfptpd and can be enabled/disabled using the 
configuration file option ptp_network_mode. sfptpd does not support unicast 
negotiation. 


Hybrid-no-fallback 


Using this mode sfptpd will always stay in hybrid mode, sending unicast Delay- 
Request messages. 


Hardware timestamps 


@ 


NOTE: Features described in this section are not available with the SFN6322F 
adapter. Customers using the SFN6322F, ignore this section. 


sfptpd can be used to enable hardware timestamping of all packets (to the Linux 
kernel) on specified interfaces. Interfaces are identified as a list using the following 
configuration file option: 


timestamping interfaces [<name | mac-address | *>] 

e To timestamp all received packets on all interfaces: 
timestamping interfaces * 

e To timestamp all received packets on eth2 and eth3: 
timestamping interfaces eth2 eth3 


If hardware timestamping is required only for PTP packets, there is no need to 
enable this parameter. 


Hardware timestamps enable/disable 


To enable/disable hardware timestamping of all received network packets after 
sfptpd exits, use the following configuration file option: 


timestamping disable on_exit [<off | on>] 
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Hardware timestamps (kernel/Onload) 


Applications can recover hardware timestamps for all received packets using the 
SO_TIMESTAMPING socket option. For more details of hardware packet timestamps 
when using the kernel driver see the Solarflare Server Adapter User Guide 
(SF-103837-CD). For more details of using hardware packet timestamps when using 
OpenOnload see the Onload User Guide (SF-104474-CD). 


PTP management messages 


PTP Management ‘GET’ messages are supported from sfptpd v2.2.1. These 
messages allow a network management node to retrieve PTP clock data from other 
PTP nodes in the network. Table 11 includes all supported managementld types. 


Table 11: Management messages 


managementld Value 

MGMT_ID_NULL 0x0000 
MGMT_ID_CLOCK_DESCRIPTION 0x0001 
MGMT_ID_USER_DESCRIPTION 0x0002 
MGMT_ID_DEFAULT_DATA_SET 0x2000 
MGMT_ID_CURRENT_DATA_SET 0x2001 
MGMT_ID_PARENT_DATA_SET 0x2002 
MGMT_ID_TIME_PROPERTIES_ DATA_SET 0x2003 
MGMT_ID_PORT_DATA_SET 0x2004 
MGMT_ID_PRIORITY1 0x2005 
MGMT_ID_PRIORITY2 0x2006 
MGMT_ID_DOMAIN 0x2007 
MGMT_ID_SLAVE_ONLY 0x2008 
MGMT_ID_LOG_ANNOUNCE_INTERVAL 0x2009 
MGMT_ID_ANNOUNCE_RECEIPT_TIMEOUT 0x200a 
MGMT_ID_LOG_SYNC_INTERVAL 0x200b 
MGMT_ID_VERSION_NUMBER 0x200c 
MGMT_ID_TIME Ox200f 
MGMT_ID_CLOCK_ACCURACY 0x2010 
MGMT_ID_UTC_PROPERTIES 0x2011 
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Table 11: Management messages (continued) 


managementlid Value 


MGMT_ID_TRACEABILITY_PROPERTIES 0x2012 


MGMT_ID_DELAY_MECHANISM 0x6000 


MGMT_ID_LOG_MIN_PDELAY_REQ_INTERVAL 0x6001 


Management messages are disabled by default. To enable sfptpd support for GET 
messages, set the following option in the PTP configuration file. 


ptp_mgmt_msgs read-only 


For a detailed description of Management Messages and the managementld values, 
refer to the Std 1588™-2008, /EEE Standard for a Precision Clock Synchronization 
Protocol for Networked Measurement and Control Systems. 


UTC offset 


The UTC offset (TAI - UTC) is conveyed to PTP slave clocks from a PTP master clock 
within the Sync or FollowUp messages. The UTC valid flag (UTCV) indicates whether 
the UTC offset time is considered valid by a master device and therefore should be 
used by a PTP slave device. 


In the sfptpd configuration file, the ptp_utc_valid_handling option identifies 
the action taken with the UTC offset value by sfptpd. 


Table 12: UTC handling 


Value Description 

default If UTCV is set, use UTC, otherwise do not. 

ignore Ignore the UTC valid flag. Always apply the UTC offset. 

prefer Prefer master clocks that set the UTCV flag above those that do 
not. 

require Do not accept PTP messages from master clocks that do not set 


the UTCV flag. 


override Ignore the UTC valid flag and UTC offset value set by the master 
<seconds> clock. Use only the value set by the override option. 


CAUTION: The user must ensure the override seconds value is 
Z \ set to the correct UTC offset. The UTC offset increments on the 
occasion of each leap second. 
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Access control lists 


Access control lists (ACLs) restrict the set of network addresses from which sfptpd 
will accept certain types of message or request. There are ACLs to control access to 
the following features: 


e  PTP timing messages 
e PTP management messages 
e —PTP monitoring requests 


From the PTP configuration file, access is controlled through separate allow and 
deny lists of IP addresses. A further configuration option dictates the order in which 
allow and deny lists are evaluated. 


The form of the configuration options is: 
e = ptp_<acl-type>_acl_allow defines the ‘allow’ list. 
e = ptp_<acl-type>_acl_deny defines the ‘deny’ list. 


e ptp_<acl-type>_acl_order defines the priorities of the two lists. 


Default behaviour 


With no ACL options specified, all messages or requests are accepted. 


Allow-Deny order 


If ‘allow’ or ‘deny’ lists are defined but no order is specified then ‘allow-deny’ order 
is assumed. This policy can be specified explicitly: 


ptp_timing acl_order allow-deny 


The default policy for the allow-deny order is that all messages or requests are 
denied. 


The ‘allow’ list defines networks and hosts that are permitted to communicate and 
the ‘deny’ list defines networks and hosts that will nevertheless be excluded. It 
makes sense for the deny list to be more specific than the allow list. 


ptp_timing acl_allow 172.16.128.0/21 

ptp_timing_acl_deny 172.16.128.48/32 172.16.128.47/32 

In the above example, the first line identifies a subnet from which, exclusively, PTP 
timing messages will be accepted. The second, optional, line identifies that access 
from the two specified hosts will be denied, despite being in the ‘allow’ list. These 


could perhaps be VPN end-points, DMZ hosts, or hosts running customer 
applications. 


G) NOTE: Note: prior to sfptpd v3.2.1, this ordering was specified by the deprecated 
deny-permit form. The new behaviour is consistent with other Internet software. 
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Deny-Allow order 
This policy is introduced with: 
ptp_timing acl_order deny-allow 


The default policy for the deny-allow order is that all messages or requests are 
accepted. 


The ‘deny’ list defines networks and hosts that will be disallowed and the ‘allow’ list 
defines networks and hosts that will nevertheless be permitted to communicate. It 
makes sense for the allow list to be more specific than the allow list. 


ptp_timing acl_deny 172.16.128.0/21 
ptp timing acl allow 172.16.128.48/32 172.16.128.47/32 


In the above example, the first line identifies a subnet from which PTP timing 
messages will be denied. The second, optional, line identifies two specific hosts that 
will be allowed to send timing messages, despite being in the ‘deny’ list. These hosts 
could perhaps be company infrastructure that is on a subnet used as a DMZ for 
visiting vendors. 


G) NOTE: Note: prior to sfptpd v3.2.1, this ordering was specified by the deprecated 
permt-deny form. The new behaviour is consistent with other Internet software. 


Outlier removal 


The sync module scans incoming values for outliers, where the offset from the 
master differs significantly from other previously received values. These outliers can 
be partially or completely removed, so they do not feedback into the time 
synchronization: 


e Outliers are detected using methods such as Peirce’s criterion. 

e The number of values stored by using the outlier_filter_size option: 
- the default value gives the best accuracy of outlier detection 
- reducing this value lowers overheads, but also reduces accuracy. 


e The amount by which outliers feedback into the time synchronization is 
controlled using the outlier_filter_adaption option: 


- the default value of 1 means that outliers are used unchanged 
- reducing this value makes outliers have less effect 


- reducing this value to 0 so outliers have no effect is not recommended. 
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7 PPS Sync Module 


7.1 Description 


Using the PPS sync module, the sfptpd slave receives time-of-day from NTP, and 
also receives a 1PPS signal. The slave runs an NTP client which sfptpd can both read 
and control. 


PPS pulses are received from a Solarflare PTP adapter with a PPS interface, sfptpd 
selects the precision clock on the adapter as the LRC. 


sfptpd periodically polls the host server NTP client to provide the time of day to the 
LRC. Thereafter sfptpd synchronizes the LRC to the 1PPS pulse, and keeps all other 
clocks (including the system clock) synchronized with the LRC. 


7.2 Configuration options 


The PPS sync module has instance-specific options that apply only to a single 
instance of the module. 
Instance-specific configuration options 


Table 13 shows the instance-specific configuration options for the PPS sync module. 
These must be in a section labeled with the instance name. 


Table 13: Instance-specific configuration options for the PPS sync module 


Option Parameters Description 

interface <interface-name> Specifies the name of the interface that PPS 
should use. 

priority <number> Relative priority of sync module instance. 


Smaller values have higher priority. 


Default: 128 


sync_threshold <number> Threshold in nanoseconds of the offset from the 
clock source over a 60s period to be considered 
in sync (converged). 


Default: 1us (hardware timestamping) 
Default: 100us (software timestamping) 


See also the local_sync_threshold option in 
Table 2 on page 33. 
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Table 13: Instance-specific configuration options for the PPS sync module (continued) 


Option 


master_clock_class 


Parameters 


locked | holdover | 
freerunning 


Description 
Master clock class. 


Default: locked 


master_time_source 


atomic|gps|ptp|ntp 
oscillator 


Master time source. 


Default: GPS 


master_accuracy 


<number>|unknown 


Master clock accuracy in ns, or unknown. 


Default: unknown 


pps_delay <number> PPS propagation delay in nanoseconds. 
pid_filter_p <number> Secondary servo PID filter proportional term 
coefficient. 
Default: 0.05 
pid_filter_i <number> Secondary servo PID filter integral term 


coefficient. 


Default: 0.001 


outlier_filter_type 


disabled | std-dev 


Specifies the filter type to use to reject outliers: 
e —_ disabled specifies no filtering 


e — std-dev applies filtering based on a sample's 
distance from the mean, expressed as a 
number of standard deviations. 


Default: std-dev 


outlier_filter_size 


<number> 


Number of data samples stored in the filter. For 
std-dev type the valid range is [5,60]. 


Default: for std-dev is 30. 
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Table 13: Instance-specific configuration options for the PPS sync module (continued) 


Option Parameters Description 
outlier_filter_ <number> Controls how outliers are fed into the filter: 
adaption 


e —avalue of 0 means that outliers are not fed 
into the filter (not recommended) 


e avalue of 1 means that each outlier is fed 
into the filter unchanged 


e values between result in a portion of the 
value being fed in. 


Default: 1.0 


fir_filter_size <number> Number of data samples stored in the FIR filter. 
The valid range is [1, 128]. 


e —avalue of 1 means that the filter is off 


e higher values will reduce the adaptability of 
PTP but increase its stability. 


Default: 4 


7.3 Default configuration file 


This section describes the default configuration file that use PPS as the only sync 
module, or as the principal module. See Example configuration files on page 37. 


pps_slave.cfg 


This file shows how to configure sfptpd to use the PPS sync module: 

e —aPPS sync module is instantiated using the sync_module option 

e other options are commented out: 
- if no changes are made, the NIC clock is used as the reference 
- configure the NTP options to use the NTP daemon as the reference. 
- configure NTP authentication as perNTP authentication on page 74. 


- uncomment and update the PPS options such as pps_ delay to refine the 
PPS performance. 


7.4 Constraints 


The PPS sync module only works on PTP-enabled Solarflare adapters with a PPS kit 
fitted. Only one sync_module should be created for each PPS interface. 
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8 NTP Sync Module 


8.1 Description 


The NTP sync module allows sfptpd to use the NTP daemon as the local reference 
clock. 


G) NOTE: NTP authentication is necessary to allow sfptpd to control the NTP client. 


8.2 Sync characteristics 


This sync module provides accuracy that is limited by the quality of the NTP 
timesource, and the reliability of NTP traffic over the network. 


NTP authentication 


Before sfptpd can query the local NTP client, it is necessary to setup symmetric key 
authentication parameters in the NTP daemon configuration files and in the sfptpd 
configuration file. 


Identical authentication integer and string value must appear in these files. The 
integer value is any positive integer. 


The maximum length of the authentication string for an M type key (MD5) should 
31 ASCII characters, but this may not be supported on earlier NTP versions. Some 
systems support up to a maximum 20 character string. Special chars should be 
escaped with a \ symbol. There can be no spaces in the authentication string. 


In the following example we use the following authentication values: 


5 abcdefghijklmnopaqrst 
1 Inthe sfptpd config file: 


[ntp_fallback] 
ntp_key 5 abcdefghijklmnopqrst 


In the above example ntp_fallback is the name of the NTP sync module instance. 


2 = Inthe /etc/ntp.conf: 


# Key file containing the keys and key identifiers used when operating 
# with symmetric key cryptography. 
keys /etc/ntp/keys 


# Specify the key identifiers which are trusted. 

trustedkey 5 

# Specify the key identifier to use with the ntpdc utility. 
requestkey 5 
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In the above example, the integer value is the same value as specified in the sfptpd 
config file. 


3 In the /etc/ntp/keys file: 


# id type key 
5 M abcdefghijklmnopqrst 


In the above example, the integer value and string are identical to the values 
specified in the sfptpd configuration file. 


Fail to start sfptpd with NTP 


If sfptpd fails to start in the NTP fallback mode and messages similar to the following 
are observed in the sfptpd messsage_log/stats_log: 


warning: ntpclient: mode6: failed to set NTP daemon system flags, Permission denied 
error: ntp: failed to disable NTP clock control 


1 = Check the authentication integer value and string value are consistent in all 
files, 


2 ‘Try with maximum 20 ASCII character string (and no special characters). 


3. ~~ Add the following to the /etc/ntp.conf file: 
enable mode7 


(There is no space between mode and 7) 


4 ‘Restart the NTP service. 
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8.3 Configuration options 


The NTP sync module has instance-specific options that apply only to a single 
instance of the module. 


Instance-specific configuration options 


Table 14 shows the instance-specific configuration options for the NTP sync module. 
These must be in a section labeled with the instance name. 


Table 14: Instance-specific configuration options for the NTP sync module 


Option Parameters Description 


priority <number> Relative priority of sync module instance. 
Smaller values have higher priority. 


Default: 128. 


sync_threshold <number> Threshold in nanoseconds of the offset from the 
clock source over a 60s period to be considered 
in sync (converged). 


Default: 10000000ns. 


See also the local_sync_threshold option in 
Table 2 on page 33. 


ntp_poll_ interval <number> Specifies how often the NTP daemon will be 
polled, in seconds. 


Default: 1 


ntp_key <key-id> <key-value> NTP authentication key ID and value. 


This is used to authenticate control requests with 
the NTP daemon: 


e = The key ID is an integer. 


It must be identified in the NTP 
configuration file, in the trustedkey and 
requestkey lists. 


e = The key value is a plain text ASCII string up 
to 31 characters long (i.e. an “m” key as 
defined by ntpd). 


It must appear in the /etc/ntp/keys file, 
identified by the key ID. 


There should be NO spaces in the 
authentication string. 


See NTP authentication on page 74. 
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8.4 Default configuration file 


This section describes the default configuration file that use NTP as the only sync 
module, or as the principal one. See Example configuration files on page 37. 


Some other configuration files use NTP as a fallback if the principal module fails. For 
example, see: 


e  ptp_master_ntp.cfg on page 63 
e  ptp_slave_ntp_fallback.cfg on page 62 
e pps_slave.cfg on page 73. 


G) NOTE: For descriptions of the remaining files in the config directory, see the 
chapters describing the other sync modules. 


ntp.cfg 


This file shows how to configure sfptpd to use the NTP daemon as the local 
reference clock: 


e —ansingle NTP sync module is instantiated using the sync_module option 


e —_ the key-id and key-value are set for authenticating with NTP using the ntp_key 
option. 


8.5 Constraints 


NTP has various performance limitations: 


e The ntpd daemon disciplines only the system clock, and sfptpd disciplines the 
remainder of the clocks from the system clock. 


e The ntpd daemon tends to slew the system clock rapidly for a short period of 
time every few minutes (e.g. 20). 


The result can be periodic errors between the system and NIC clocks, while the 
adapter clocks catch up with this rapid change in the system clock. The 
behavior will get better over time. For example, once the host has been up for 
24 hours the NTP adjustments will be much smaller. 


e The accuracy of NTP can vary massively. Solarflare recommend only using this 
mode with a GPS-connected NTP server. 
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9 Freerun Sync Module 


9.1 Description 


The Freerun sync module allows sfptpd to use a NIC clock or system clock as the 
local reference clock. This Solarflare adapter clock is a Stratum 3 compliant 
oscillator, accurate to 0.37 PPM per day. 


9.2 Sync characteristics 


This sync module provides low drift, but no synchronization to an external clock. 


9.3 Configuration options 


The Freerun sync module has instance-specific options that apply only to a single 
instance of the module. 


Instance-specific configuration options 


Table 15 shows the instance-specific configuration options for the Freerun sync 
module. These must be in a section labeled with the instance name. 


Table 15: Instance-specific configuration options for the Freerun sync module 


Option Parameters Description 


interface <interface-name> Specifies the name of the interface hosting the 
local reference clock. 


priority <number> Relative priority of sync module instance. 
Smaller values have higher priority. 


The default is 128. 


If the priority is specified in a generic section - outside of a specific (named) sync instance, 
the priority will apply to all sync modules of the same type e.g. 


sync_module ntp namedntp1 namedntp2 


[ntp] 
// this applies to all NTP type sync module instances 
priority 128 
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[namedntp1] 


// this applies only to this instance - and takes priority over any generic value 


priority 127 


[namedntp2] 


// this applies only to this instance - and takes priority over any generic value 


priority 126 


9.4 Example configuration files 


This section describes the example configuration files that use Freerun as the only 
sync module, or as the principal one. See Example configuration files on page 37. 


G) NOTE: For descriptions of the remaining files in the config directory, see the 
chapters describing the other sync modules. 


freerun.cfg 


This file shows how to configure sfptpd to use the NIC clock as the local reference 
clock: 


e aFreerun sync module is instantiated using the sync_module option. 


9.5 Constraints 


The Freerun sync module is not compatible with VLANs, because there is no 
incoming data. 
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10 Sfptpd Operation 


10.1 Adjusting the System Clock 


sfptpd uses CLlock_adjtime( ) to control/adjust the system clock. Users 
monitoring this system call may wish to add an exception to prevent sfptpd entries 
flooding any monitoring logs. 


10.2 PTP Multicast Groups and Ports 


The PTP protocol uses the following multicast addresses: 
224.0.1.129 224.0.0.107 


sfptpd will join these groups and periodically renew group interest using the IGMP 
protocol. 


Use the Linux netstat -ng command or the following command to identify multicast 
groups joined on the sfptpd interface: 


# cat /proc/net/igmp 


Idx Device : Count Querier Group 
1 lo d 1 v3 

010000EO 1 
2 etha : 1 V2 

010000E8 1 
10 virbre@ : 1 v3 

010000E8 1 
15 eth4 : 3 V2 

6BOG00EO 2 

810100EO 2 
81 01 00 EO 6B 00 00 EO 
129 1 @ 224 107 @ @ 224 


PTP only uses UDP ports 319 and 320. To capture only PTP traffic use tcpdump in the 
following format: 


# tcpdump -i <sfptpd interface> dst port 319 or dst port 320 [-wptp.pcap] 
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10.3 CPU affinity - sfptpd threading model 


sfptpd is a multi-threaded application. The singular task of the main sfptpd process 
is to spawn a number of child processes which carry out the sfptpd processing tasks. 
The main sfptpd thread does no further work and, once started, will ignore attempts 
to affinitize to another CPU core. 


At startup the following command will ensure that sfptpd - and its child threads will 
run on CPU core 1. 


# taskset -C 1 ./sfptpd -i <interface> -fconfig/ptp_slave.cfg 


Thereafter attempts to move the sfptpd process to a different thread will fail 
because the main thread is doing no work - so takes not notice of affinity changes. 


But the child/worker threads can be directed to other CPU cores while sfptpd is 
running. It is strongly recommended that all workers are affinitized to the same CPU 
core. 


The following example (ptpthreads) script can be used as a simple shell script to 
identify sfptpd worker threads and affinitize all to a specific CPU core: 


#!/bin/sh 

# 

# usage: ptpthreads 2 
# 

DESIRED_CPU=$1 


SFPTPD_PIDS=$(ps -T -p $(pidof sfptpd) | grep -v SPID | awk '{print $2}') 
for i in ${SFPTPD_PIDS[@]}; do 

taskset -c -p $DESIRED_CPU $i; 
done 


# Confirm changes for sfptpd threads which are busy: 
ps -Leo cpuid,lastcpu,pid |grep $(pidof sfptpd) 


10.4 Comparing NIC and System Clocks. 


The phc_ctl utility is part of Linux PTP. If linuxptp is installed on the server, phc_ctl 
can be used to set the Solarflare adapter clock and compare the time offset against 
the system clock. 


Get System Clock Time 
# phc_ctl CLOCK_REALTIME get 


Get NIC clock Time 
# phc_ctl <interface> get 


Compare System and NIC clocks 
# phc_ctl /dev/ptpN cmp 
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Set NIC clock from System clock 
# phc_ctl /dev/ptpN set 


The NIC clock can be specified by the sfptpd interface or by device number. 


10.5 PPS Output 


The PPS output is not latched to the PPS input. The output pulse is triggered by the 
start of second period on the adapter clock. When comparing the PPS output to an 
external timesource, the adapter clock should be closely synchronized to the 
external timesource. 


PPS ‘chaining’ is not a supported mode of operation. 


10.6 Saved clock frequency correction data 


For each Solarflare adapter clock and for the server system clock, sfptpd will save 
the current frequency correction value every 60 seconds. If the server is rebooted 
or if sfptpd is restarted, the last saved value is used to continue clock frequency 
adjustment rather than revert to a zero value which would delay re-synchronization 
to an external master clock (or re-synchronization to the LRC if the adapter clock is 
not the active clock receiving from the remote master clock). 


Frequency correction files are saved in /var/lib/sfptpd. Refer to Table 6 on 
page 46 for details. 


10.7 sfptpd in operation 


The following charts show sfptpd performance when the Solarflare adapter is 
configured in PTP slave mode to a third party Grandmaster clock via a network 
switch. In this example Ethernet interface 4 (eth4) is the interface receiving PTP 
packets and sfptpd is disciplining the system clock. 


‘[ptp-gm->eth4] offset’ on PTP slave dellr610i (NTP off) 


Offset (nanosec) 


1 1 1 1 1 
800 1000 1200 1400 1600 1800 
Elapsed Time (seconds) 


Figure 6: Offset of adapter’s clock to the PTP master 
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‘Leth4->system] offset’ on PTP slave dellr610i (NTP off) 
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Figure 8: Offset of the server’s system clock to the PTP master 


The charts below identify sfptpd performance when two Solarflare PTP adapters 
are present and configured in a bonded interface in a PTP slave server. Arrows on 
the chart identify what happens during bond failover and failback events. 


The upper chart shows the consistent offset of the server’s system clock from the 
clock on the active port during repeated bond failover and failback events. For the 
same period, the middle chart shows the offset of the active adapter clock from the 
external grandmaster during the failover and failback of the bonded ports. For the 
same period the lower chart shows the offset of the server system clock from the 
external PTP master clock. 
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NIC to System Clock Synchronization 
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Figure 9: Bonding failover performance 
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10.8 Handling of leap seconds 


The timestamps in PTP messages received from a master clock are in TAI (atomic) 
timescale. When a PTP message is received, sfptpd will first convert the atomic 
timestamp to a UTC timestamp for comparison with the internal UTC clock time. 
When a positive leap second is inserted in the UTC timescale, an additional second 
is inserted which effectively causes the UTC time to step backwards by one second. 
Following the leap second the UTC offset, currently 37 seconds, becomes 38 
seconds. 


UTC time =(TAI time - UTC offset) 


The action applied when a leap second occurs will depend on the sfptpd 
clock_control configuration option enabled in the sfptpd configuration file. 


The clock_control option determines the step/slew action applied to the Local 
Reference Clock (LRC) at all times, including when a leap second occurs. (The LRC is 
the clock that is currently being disciplined by sfptpd.) 


Solarflare sfptpd is able to slew the clock at a maximum rate of 1 million ppb (1 part 
in a 1000) so slewing will take approximately 20 minutes to recover 1 second. The 
slew rate is not configurable. 


Corrective action is applied to the LRC. Other clocks in the server will be 
synchronized with the LRC. 


Table 16: Leap second action 


Clock_control Option Leap second Action 


slew-and-step Default. Suspend clock synchronization, step the clock 
currently being disciplined backwards by one second 
and then recommence normal synchronization when 
the next announce message is received. 


step-at-startup Slew the clock to correct for leap second insertion. 

no-step Slew the clock to correct for leap second insertion. 

no-adjust Do not adjust clock. 

step-forward Will not step the clock when a positive leap second 
occurs, but will slew the clock to correct for leap second 
insertion. 
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10.9 Daemon control mechanism 


sfptpdctl asa utility that can be used by to trigger housekeeping operations in the 
daemon. 


Control commands 


Table 17 shows the control commands: 


Table 17: Control commands 


Command String for ctrl proto equiv. signal Description 

Rotate logs logrotate SIGHUP Closes and reopens all log files 

Step clocks stepclocks SIGUSR1 Causes all clocks to be advanced 
or retarded to the time of the 
parent clock 

Exit daemon exit SIGINT Causes the application to exit 

SIGTERM without error 
Select instance selectinstance=<instance-name> — Selects the sync module instance 


when in manual selection mode. 
See Manual selection on page 20. 


Control protocol 
The client connects to the control server address. The version of the protocol is 
included in the address. The current version is 1 and the address is: 
/var/run/sfptpd-control-v1.sock 


The client will only succeed in connecting to the server if it has write permissions to 
the server socket and the server socket is created using the umask the server 
process inherited, therefore normally it will be necessary for the client to be run as 
root. 


A command is sent as plain text in a Unix Domain datagram, for example: 
<command string> 


There is no return channel and it does not matter to what address, if any, the client's 
socket is bound. 


Two example implementations of the client are included, both of which are suitable 
to invoke from the command line or to incorporate in custom scripts or operations 
code. 
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Standalone client 


A standalone sfptpdct1 client is supplied: 

e Inthe tarball distribution, this client is located in the install directory. 
e Inthe RPM distribution, this client is located in /usr/sbin. 

To run, invoke with the desired control command: 

sfptpdctl <command> 

To select a sync module instance named ptpdomain1: 


sfptpdctl selectinstance=ptpdomain1 


Standalone client source code 
C source code for the standalone client is supplied as sfptpdctl.c: 


e In the tarball distribution, this source code is located in the examples 
subdirectory of the install directory. 


e Inthe RPM distribution, this source code is located in the /usr/share/doc/ 
packages/sfptpd/examples directory. 


To build, run make in that directory. 


Python client 


A simple Python client is supplied as sfptpdct1. py: 


e —_ Inthe tarball distribution, this client is located in the examples subdirectory of 
the install directory. 


e = Inthe RPM distribution, this client is located in the /usr/share/doc/ 
packages/sfptpd/examples directory. 


System log rotation 


The sfptpdctl logrotate command causes the sfptpd daemon to close and reopen 
the log file, creating the file if it does not already exist. 


G) NOTE: This action does not rename the log file. 


The sfptpdctl logrotate command could be called from the postrotate section of a 
standard linux logrotate configuration file. For example: 


postrotate 
<path>/sfptpdctl logrotate || true 
endscript 


An alternative action would be to move/rename the current log file before invoking 
the sfptpdctl logrotate command. 
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Stepping Clocks 


All clocks in the sfptpd PTP slave server can be stepped at any time using the 
sfptpdctl utility. 


Clocks will be stepped, forwards or backwards, into the same second period as the 
parent clock. 


When clocks are stepped, entries similar to the following are generated in sfptpd 
message log/stats_log: 


# sfptpdctl stepclocks 


notice: received 'stepclocks' control command: stepping clocks to current 
offset 

info: clock phc@: applying offset @.288991267 seconds 

info: clock phc1: applying offset 1.376286052 seconds 

info: clock system: applying offset @.380292621 seconds 
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11 Options - Detail 


This section provides detail to some of the sfptpd configuration file options. 


11.1 Best Master Clock Discriminator 


sfptpd version 3.3.0.1007 provides an option to disqualify PTP master clocks which 
advertise a time differing by more than a specified threshold from a selected 
timesource. 


A PTP slave can disqualify a master clock based on the received timestamp in the 
following messages: 


e SYNC message (in the case of 1-step synchronization) 
e FOLLOWUP message (in the case of 2-step synchronization) 


When a master clock is disqualified it is excluded from the synchronization process 
and is removed from the sfptpd foreign master records until such time it reports 
time offset values below the threshold value. 


This feature is primarily designed to use NTP as the discriminator against multiple 
master clocks within the same PTP domain. 


To use NTP as the discriminator, configure both PTP and NTP sync_modules in the 
sfptpd config file. 


Set the NTP sync_module priority lower than the PTP priority to ensure that NTP is 
used as the discriminator, but not selected ahead of any PTP clock as the 
synchronizing timesource. 


Usage: 
bmc_discriminator <CLOCK> <THRESHOLD> 
e CLOCK can be a sync instance name or clock name 


e THRESHOLD is a microsecond value. 


Example configuration file: 


This example contains only parameters to demonstrate the bmc_discriminator 
feature using NTP as the discriminator- refer to example configuration files for other 
parameters. 

[general] 


sync_module ptp ptp1 ptp2 
sync_module ntp ntp1 


[ptp1] 
ptp_mode slave 
ptp_domain 101 
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priority 10 


[ptp2] 
ptp_mode slave 
ptp_domain 101 
priority 20 


[ptp] 
interface eth1 
bmc_discriminator ntp1 1 


[ntp1] 
ntp_key 8 bmctest 
priority 255 


G) NOTE: The server system clock or a NIC clock could be selected as the CLOCK 
discriminator, but these are untested scenarios. 


11.2 Epoch Guard 


This feature is a guard against a NIC clock that resets to the UTC epoch (01 Jan 1970). 
This can occur, for example, if the NIC is reset or if an external master clock 
malfunctions directing the downstream slaves to show a current time at or near the 
UTC epoch. 


This option is enabled by default, where the default action is to raise an alarm and 
prevent the system clock from being synchronized to the affected NIC clock. 


e — alarm-only 


Raise an alarm when the NIC clock is near epoch. Does not prevent system clock 
from being synchronized to the affected NIC clock. 


¢ — prevent-sync (default) 


Raise an alarm and prevent the system clock from being synchronized to the NIC 
clock when the NIC clock is near the epoch. 


e ~—_correct-clock 


Raise an alarm and set the affected NIC clock to the system clock time. 


Usage: 
Enable the option in the sfptpd configuration file. 


epoch_guard [alarm-only | prevent-sync | correct-clock] 


G) NOTE: This option does not consider the clock_control options. 
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11.3 PTP Network Mode 


Using the default hybrid mode, at startup, sfptpd will send three unicast Delay- 
Request messages to the upstream master clock. If the master fails to respond to 
these unicast messages, sfptpd will automatically switch to full multicast mode and 
thereafter send multicast Delay-Request messages. Once it has switched to 
multicast mode, sfptpd does not revert to unicast mode. 


The ptp-network-mode option allows the following modes: 
e — hybrid (default) 


At startup, sfptpd will send 3 x unicast Delay-Request messages to the upstream 
master. If the master clock fails to respond, sfptpd will revert to multicast mode. 


e = multicast 
Always send Delay-Request messages as multicast messages. 
e — hybrid-no-fallback 


Always send Delay-Request messages as unicast messages - do not revert to 
multicast. 


Usage: 
Change the option in the sfptpd configuration file. 
ptp-network-mode [hybrid | multicast | hybrid-no-fallback] 


11.4 Chronyd 


sfptpd from version 3.3.0.1007 supports a limited use case with sfptpd running 
alongside chronyd. 


Unlike NTP, chronyd does not expose an API that can be used to prevent it from 
controlling the system clock. Therefore, to allow chronyd and sfptpd to coexist, the 
following option must be enabled in the sfptpd configuration file: 


clock_readonly system 


The system clock is then controlled and adjusted by chronyd, but only read by 
sfptpd. 


Solarflare recommend that chronyd is disabled and standard NTP installed when 
NTP functionality is required. 
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11.5 Clock Readonly 


The clock_readonly option is a list of clocks which sfptpd should never synchronize 
or adjust. 


This option can be used, for example, to prevent sfptpd from adjusting the system 
clock when running alongside chronyd. Chronyd in such a configuration will 
synchronize the system clock. 


Usage: 
clock_readonly [name | mac-address | clock-id | interface] 


Specifies a set of clocks that sfptpd should never step or slew, under any 
circumstance. 


)\ CAUTION: Use with care to avoid unexpected behaviours. 
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12 PTP on VMware ESXi 


12.1 Introduction 


This section identifies the requirements and configuration procedures for running 
PTP on one or more Virtual Machines (VM) hosted on VMware ESXi. 


sfptpd will synchronize the Solarflare adapter clock and system clocks from multiple 
VMs to an upstream PTP master clock. 


12.2 How it works 


PTP messages are received over a network from an upstream PTP Grandmaster 
Clock and delivered via a PCle Virtual Function (VF) to the sfptpd daemon running 
on one or more Virtual Machines. 


One VM is selected to be the Local Master Clock synchronizing the Solarflare 
adapter clock with the upstream PTP Grandmaster Clock and synchronizing its own 
OS system clock to the adapter clock. 


Other VMs will run as PTP slave clocks using sfptpd to synchronize their own OS 
system clocks with the adapter clock. Each VM must have at least one SR-IOV 
enabled VF from the Solarflare adapter in the host. 


VM Local Master VM Slave 


System Clock 


VM Slave 


System Clock 


PTP Grandmaster 


Ref Clock 


Solarflare NIC (Local Reference Clock) 
as) . 1 ESXi host 


)\ CAUTION: There can only be one Local Master Clock on an ESXi host. 
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12.3 Requirements 


Solarflare drivers and user guides are available from https://support.solarflare.com/ 


Software/Driver Packages 


Native Driver for ESXi host: 
Supports the SFN8000 series and X2 series adapters. 


Part Number Minimum version 
SF-118824-LS version 5, driver version 2.3.2.0000 
SF-120055-LS version 4, CIM-Provider version 2.1.0.24 


Legacy Driver for ESXi host: 
Supports the SFN5000, SFN6000, SFN7000 and SFN8000 series adapters. 


Part Number Minimum version 
SF-111981-LS version 11, driver version 4.10.10.1001 
SF-115711-LS version 3, CIM-Provider version 2.0.0.3 


ESXi VM guest packages: 


Part Number Description 


SF-108910-LS Solarflare Enhanced PTP Daemon (sfptpd) 


- install in each guest VM. 


SF-103848-LS Solarflare adapter driver for Linux (src RPM) 


- install in each guest VM. (or use Onload) 


CAUTION: The user must ensure that the correct driver and compatible CIM 
Provider packages are installed to support the model of adapter. 


Issue 8 


© Copyright 2020 Xilinx, Inc 94 


&~ Solarflare Enhanced PTP User Guide 
SOLARFLARE® 
A XILINX COMPANY PTP on VMware ESXi 
ESXi Host 
e The Solarflare net driver must be installed on the host. Install either the legacy 
driver or the native driver depending on the adapter model. 
e The CIM provider must be installed on the host. 
e Host and adapter should be configured for SR-IOV. 
VM Guest 
e —_ Solarflare sfptpd is supported on Linux guests (Linux variant OS) only. Refer to 
Hardware and Software Support on page 9. 
e The Solarflare net driver (SF-103848-LS) must be installed in the guest OS. 
Onload users can use the sfc driver from the Onload installation. 
e — Solarflare sfptpd will run in each VM. 
e — NTP, chronyd, timesyncd and timedated should NOT be running in the guest OS. 
SRIOV 


Each VM must have at least one VF connected as an SR-IOV network adapter. For 
details of SR-IOV configuration on ESXi - refer to the Solarflare Server Adapter User 
Guide (SF-103837-CD). 


Adapter PTP Key 


The Solarflare adapter must have a PTP/HW timestamping activation key installed. 
There is currently no way to add or check the activation key from the ESXi host. 


To add/check for the PTP activation key on a Linux server, use the sfkey utility from 
the Solarflare Linux Utilities package (SF-107601-LS): 


# sfkey 


p1p1,p1p2: @@0F53650950 


Product name Solarflare XtremeScale X2522-25G Adapter 
Serial number 252221402110182318500104 
Installed keys Plus 


p2p1,p2p2: @@QF535CFFAQ 


Product name Solarflare Flareon Ultra 800@ Series 10G Adapter 
Serial number 852200213000173117100590 
Installed keys Onload, PTP 


An adapter with a Plus key includes the PTP key. 
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12.4 Configuration 


Configuration sequence 


e Configure host server SR-IOV. 
- refer to the Solarflare Server Adapter User Guide (SF-103837-CD). 


e ESXi Host - Adapter Driver on page 96. 

e ESXi Host - CIM Provider on page 97. 

e Prepare SRIOV on Solarflare Adapter on page 98 
e VM Driver Install on page 98. 

¢ Connect VF to the VM on page 98 

e Install sfptpd in the VM on page 98. 

e Enable VM PTP Privileges on page 99. 

e Run sfptpd on page 101. 


G) NOTE: On the ESXi host, the legacy Solarflare adapter driver module name is sfc. 
The native driver module name is sfvmk. 


ESXi Host - Adapter Driver 


The Solarflare net adapter driver must be installed on the ESXi host. Version 
numbers are different between legacy and native driver VIBs. 


Install the VIB through the host CLI 
esxcli software vib install -v <absolute PATH to the .vib> 


Installation Result 

Message: The update completed successfully, but the system needs to be 
rebooted for the changes to be effective. 

Reboot Required: true 

VIBs Installed: SFC_bootbank_4.10.10.1001-10EM.550.0.0.1331820 

VIBs Removed: 

VIBs Skipped: 


Identify installed/loaded driver module 


esxcli system module get -m=[sfc|sfvmk] 


Module: sfc 

Module File: /usr/lib/vmware/vmkmod/sfc 

License: GPL 

Version: Version 4.10.10.1001, Build: 1331820, Interface: 9.2 Built on: 
Jun 13 2018 

Build Type: 

Provided Namespaces: 

Required Namespaces: com.vmware.driverAPI@9.2.2.0, 
com.vmware.vmkapi@v2_2 0 0 
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Containing VIB: net-sfc 
VIB Acceptance Level: certified 
Set driver module parameters: 
Must be set for the required number of VFs the driver is to support: 
esxcli system module parameters set -m [sfc|sfvmk] -p max_vfs=N 
A maximum 63 VFs are supported per PF. 
View driver module parameters: 
esxcli system module parameters list -m [sfc|sfvmk] 
ESXi Host - CIM Provider 
The CIM Provider package includes the CIM Provider .vib and a python script 
set_vm_ptp_privilege.py. 
The CIM Provider must be installed on the ESXi host. The python script should be 
copied to a standard Linux server. 
Install the VIB through the host CLI 
esxcli software vib install -v <absolute PATH to the vib>/SFC-ESX- 
solarflare-cim-provider-2.0-0.3.vib 
Installation Result 
Message: The update completed successfully, but the system needs to be 
rebooted for the changes to be effective. 
Reboot Required: true 
VIBs Installed: SFC_bootbank_solarflare-cim-provider_2.0-0.3 
VIBs Removed: 
VIBs Skipped: 
NOTE: When a vib has been installed the ESXi host server must be rebooted. 
Verify CIM provider status 
esxcli software vib list | grep solarflare 
solarflare-cim-provider 2.0-@.3 SLF VMwareAccepted 2018-12-18 
esxcli system wbem provider list 
Name Enabled Loaded 
SLF_solarflare-cim-provider true true 
sfcb_base true true 
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Prepare SRIOV on Solarflare Adapter 


This configuration requires SRIOV configured with the sfboot or sfboot_esxi utility 
on the Solarflare adapters. 


e The adapter firmware variant must be full-feature/virtualization. 
e When using the legacy drivers, - on the ESXi host, use the sfboot utility from the 
Solarflare Linux Utilities package (SF-105095-LS). 


sfboot -i=<vmnicX> firmware-variant=full-feature switch-mode=sriov pf- 
count=1 vf-count=N 


e When using the native drivers, use the sfboot_esxi utility version 7.6.1.1000 or 
later to connect from a remote Linux server with the CIM provider installed on 
the ESXi host. 


sfboot_esxi -i<vmnicX> firmware-variant=full-feature switch-mode=sriov 
pf-count=1 vf-count=N -a "https://fully qualified server domain 
name>:5989" -u <user> -p <user password> 


e Where N is the number of VFs created. Each VM will require a separate VF. A 
maximum 63 VFs are supported per PF. 


e The host server must be rebooted following sfboot adapter changes. 


e The Solarflare adapter must have a PTP/HW timestamping activation key. 


VM Driver Install 


A Solarflare adapter driver must be installed in the guest OS. The driver can be 
installed from any of the following packages. 


e  SF-103848-LS - the source RPM 
e SF-104979-LS - the DKMS RPM 
¢ OpenOnload distribution - includes the adapter driver 


For driver installation refer to the Solarflare Server Adapter User Guide (SF-103837- 
CD). 
Connect VF to the VM 


Each VM should have a VF from the Solarflare adapter. The VF must be an SR-/OV 
network adapter. 


For SRIOV configuration in VMware, refer to the Solarflare Server Adapter User 
Guide (SF-103837-CD). 


Install sfptpd in the VM 


Refer to Download and install sfptpd on page 25 to install the sfptpd daemon. 
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12.5 Enable VM PTP Privileges 


The set_vm_ptp_privilege.py script is used to identify SR-IOV enabled adapters in 
the ESXi host and to enable PTP privileges allowing the VF connected to the Local 
Master VM to control and adjust the adapter hardware clock. 


Linux Server ESXi Host 


set_vm_ptp_privilege.py CIM Provider 


The script is run from a remote Linux server and will connect with the Local Master 
VM via the CIM Provider installed on the ESXi host. 


The following (minimum version) python components must be installed on the 
remote Linux server. 


e ~~ python 2.7.5 
e — python SDK for VMware : pyymomi 
¢ pywbem 0.12 


NOTE: The Local Master VM must be in a ‘power ON’ state. 
@ NOTE: It is only necessary to run the python script to the Local Master Clock VM. 


Get SRIOV adapter information 
Usage: 


python./set_vm_ptp_privilege.py <cmd> <server fully qualified hostname> 
<Local master virtual machine> 


cmd can be info|set-master 


Example: 


# python ./set_vm_ptp_privilege.py info mservi.domain.com vm-ptp 


Connecting to mservi.domain.com ... 
username: root 
password: 
mserv1.domain.com 
Solarflare NICs 
id: 0000:81:00.@ deviceId: @x@a@3 (SFC922@) 
id: 0000:81:00.1 deviceId: @x@a@3 (SFC9220) 
id: 0000:81:00.2 deviceId: @x1a@3 (<class> Ethernet controller) 
id: 0000:81:00.3 deviceId: @x1a@3 (<class> Ethernet controller) 
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[vm-ptp] 
power: on 
sriov ethernet devices: 
SR-IOV network adapter 1 (pt-ptp) 
pf: Automatic -0000:81:00.0 
vf: 0000:81:00.2 


Adapter deviceld: 

e  0x1a03 VF on SFN7000 or SFN8000 series adapter. 
e 0x1b03 VF on X2 series adapter. 

e  0x0a03 PF on SFN7000 or SFN8000 series adapter. 
e  0x0b03 PF on X2 series adapter. 


Set PTP privileges on the Local Master VM 


# python ./set_vm_ptp_privilege.py set-master mserv1.domain.com vm-ptp 


Connecting to serveri.domain.com ... 
username: root 
password: 
mserv1.domain.com 
Establishing connection to host:mserv1.domain.com 
Configuring mserv1.domain.com:vm-ptp to be the Local Master ... 
Current VM power state is on 
Waiting for guest operations to become ready... 
Configuring VF for Local Master mode of operation 
id: 0000:81:00.2 
devicelId: @x1aQ3 
name: <class> Ethernet controller 


Following Privileges are set currently: 
ONLOAD 


Setting PTP Privilege .. 

Following Privileges are set currently: 
ONLOAD 

PTP 


G) NOTE: It is only necessary to run the python script to the Local Master Clock VM. 
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12.6 Run sfptpd 


When there is only a single VM - sfptpd should run as the local master. When there 
are multiple VMs, one VM is the local master, all others will run the freerun.cfg. 


Local Master 


# Example configuration for a local master VM. 

# 

[general] 

sync_module ptp ptp1 

message log stderr 

stats_log stdout 

# Uncomment this line if passing more than one VF to the guest. 
#clock_list system phc@ 


[ptp1] 

ptp_mode slave 
ptp_tx_latency @ 
ptp_rx_latency @ 
ptp_network_mode hybrid 

# PTP domain, default is 0 
ptp_domain 100 


[ptp] 
# Specify the interface to use - or on command line -i argument. 
# interface ens224 


Command Line 


# ./sfptpd -i ens224 -f<path>/ptp_local_master.cfg 


Freerun VMs 


# Example configuration for synchronising a guest system clock 
# to the NIC clock 


[general] 

sync_module freerun fri 

message log stderr 

stats_log stdout 

clock_control no-step 

# Uncomment this line if passing more than one VF to the guest. 
#clock_list system phc@ 


[fr1] 


# Specify the interface to use - or on command line -i argument. 
# interface ens224 


Command Line 


# ./sfptpd -i ens224 -f<path>/freerun.cfg 
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12.7 sfptpd Output 


The following sfptpd stats_log/message_log extracts identify typical output 
generated by sfptpd. 


Local Master 


From a VM running as the Local Master, output similar to the following should be 
observed. (ens224 is the sfptpd interface (VF) in this example). 


2019-01-06 16:14:02.125089 [ptp1:gm->phc@(ens224)], offset: -15.000, freq- 
adj: 399.273, in-sync: 1, one-way-delay: 3166.000, parent-id: 
Q0e0F:53FF:fe55:aabb, gm-id: £452:14ff:fe4c:7020 


2019-01-06 16:14:02.126189 [servo@:phc@->system], offset: -15.000, freq- 
adj: 399.273, in-sync: 1 


e = The first line identifies the time offset (nanoseconds) of the adapter clock from 
the upstream master clock. 


e The second line identifies the time offset (nanoseconds) of the guest OS system 
clock from the adapter clock. 


Freerun VMs 


From a VM running the sfptpd freerun.cfg, output similar to the following should be 
observed. (ens224 is the sfptpd interface (VF) in this example). 


2019-01-06 16:14:01.125088 [servo@:phc@->system], offset: -5.312, freq- 
adj: 342.116, in-sync: 1 


2019-01-06 16:14:01.125088 warning: clock phc@(ens224): failed to set sync 
status: Operation not permitted. 


e = The first line identifies the time offset (nanoseconds) of the guest OS system 
clock from the adapter clock. 


e On the second line a warning is generated because a VF which is not the Local 
Master is not permitted to adjust the adapter hardware clock. This can be 
ignored. 


Additional log files 
sfptpd will generate additional log files in the /var/lib/sfptpd directory. 
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12.8 Checking Configuration 


ESXi Host 
e —_Use Ispci in the ESXi host to confirm that VFs are configured - examples below: 


Ispci | grep Solarflare 


Q000:81:00.0 Network controller: Solarflare SFC9226 [vmnic4] 
Q000:81:00.1 Network controller: Solarflare SFC922@ [vmnic5] 


Ispci | grep "81:00" 

0000:81:00.@ Network controller: Solarflare SFC9226 [vmnic4] 
@000:81:00.1 Network controller: Solarflare SFC922@ [vmnic5] 
Q000:81:00.2 Network controller: [PF_@.129.0 VF_@] 
Q000:81:08.3 Network controller: [PF_0.129.0 VF_1] 

Ispci | grep _VF_ 


0000:81:00.2 Network controller: [PF_@.129.0 VF_@] 
@000:81:00.3 Network controller: [PF_@.129.0 VF_1] 


e Check network topology (vswitch and port group) for VM connectivity. In the 
example below, there are two VMs: vm-ptp and vm-ptp2. Each VM has its own 
VF from the physical adapter in the host: 


¥ vSwitch topology 


@ port-ptp a Physical adapters 
VLAN ID: 0 aH §H vmnic4, 10000 Mbps, Ful 
~ Virtual Machines (2) 
G® vm-ptp 
MAC Address 00:0¢:20:42:74:55 = 
Ep vm-ptp2 
MAC Address 00:0¢:20:ab:47:88 —t | 
= 


NOTE: A VF does not use a port-group for sending/receiving traffic. The port-group 
is used only to apply networking properties such as VLAN ID. 
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VM - Local Master 
Check the /var/lib/sfptpd/topology file: 


The ptp grandmaster will identify the upstream PTP master clock 
The state is ptp-slave 


The offsets for GM to adapter and adapter to system are recorded. 


State: ptp-slave 
interface: ens224 Cens224) 
imestamping: hw 


grandmaster 
0058 :c2ff :fede:95b?/1 


9.006 ns 


i 
V 
phc# Cens224) 
O06c :29ff :fed2:7455 
i 


-4.436 ns 


Check Local Master VM SRIOV adapter configuration: 


~ Hardware Configuration 


> I cPu 41 vCPUs 
i Memory 2GB 
> <3 Hard disk 1 16 GB 
USB controller USB 2.0 
> Ml Network adapter 1 VM Network (Connected 


> Mil SR-IOV network adapter 1 


Network port-ptp (Connected 
Connected Yes 
MAC address 00:0¢:29:d2:74:55 
Pass-through (Direct-path I/O) No 

> [il Video card 8MB 

> [ll Others Additional Hardware 
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VM - Freerun 

Check /var/lib/sfptpd/topology file: 
e The state is freerun 

e The adapter clock is identified 


e = The offset of the guest OS system clock from adapter clock is recorded. 


State: freerun 


phc# (ens224) 
OBGc :29ff :feab :4788 


-2.562 ns 


e Check VM SRIOV adapter configuration: 


~ Hardware Configuration 


> fd cPU 1 vCPUs 
i Memory 2GB 
> (3 Hard disk 1 16 GB 
USB controller USB 2.0 
> Mil Network adapter 1 VM Network (Connected 


y M@ SR-IOV network adapter 1 


Network port-ptp (Connected 
Connected Yes 
MAC address 00:0¢:29:ab:47:88 
Pass-through (Direct-path I/O) No 

> [il Video card 8 MB 

> [a Others Additional Hardware 


TCPDUMP 


The standard Linux tcpdump can be used in the Linux guest to confirm that PTP 
traffic is being received by the VF: 


# tcpdump -i <sfptpd VF interface> dst port 319 or dst port 320 
(PTP only uses ports 319 and 320) 


Issue 8 © Copyright 2020 Xilinx, Inc 105 


SOLARFLARE® 


A XILINX COMPANY 


Solarflare Enhanced PTP User Guide 
PTP on VMware ESXi 


12.9 VF Statistics 


The following commands on the ESXi host identify VF availability and stats: 


Identify SRIOV NICs: 
esxcli network sriovnic list 


Name PCI Device Driver Link Speed Duplex MAC Address MTU 
vmnic4 @@00:81:00.0 sfc Up 1900@e¢e Full 00:0F:53:45:F3:30 1500 


Description 
Solarflare SFC9220 


Identify VFs on the SRIOV NIC: 


esxcli network sriovnic vf list -n vmnic4 


VF ID Active PCI Address Owner World ID 


2) true 00000:129:00.2 68953 
1 true 00@000:129:00.3 69105 


Display stats counters per VF: 
esxcli network sriovnic vf stats -n vmnic4 -v @ 


NICName: vmnic4 

Rx Broadcast Bytes: @ 
Rx Broadcast Pkts: @ 
Rx Errordrops: @ 

Rx LROBytes: @ 

Rx LROPkts: @ 

Rx Multicast Bytes: @ 
Rx Multicast Pkts: @ 
Rx Outof Bufferdrops: @ 
Rx Unicast Bytes: @ 
Rx Unicast Pkt: @ 

Tx Broadcast Bytes: @ 
Tx Broadcast Pkts: @ 
Tx Discards: @ 

Tx Errors: @ 

Tx Multicast Bytes: @ 
Tx Multicast Pkts: @ 
Tx TSOBytes: @ 

Tx TSOPkts: @ 

Tx Unicast Bytes: @ 
Tx Unicast Pkt: @ 
VFID: @ 


-n identifies the physical PTP adapter interface in the ESXi host 


-v identifies the VF ID. 
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13.1 Master NTP mode 


@ 


Using the Solarflare adapter as a PTP master clock in NTP mode 
(ptp_master_ntp.cfg), the local NTP client periodically synchronizes with a 
remote NTP server (configurable through the local NTP daemon). NTP is used to set 
the system clock and adapter clock and sfptpd keeps the system clock in sync with 
the adapter precision oscillator. 


NTP Server 


ntp client 


system time 


PTP 


Figure 10: Master NTP mode 
The sfptpd daemon synchronizes with the local NTP client every 1 second (default) 
and this is configurable in the sfptpd configuration file. 


If the master is not the active PTP master, it will by default, revert to a PTP slave 
clock. 


NOTE: When in master clock mode, sfptpd uses the UTC timescale. Ensure the 
ptp_utc_offset option in the sfptpd master configuration file is set to 0. 
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13.2 Standby master clock 


It is quite common to deploy the Solarflare adapter in master clock mode as a 
secondary/backup master when the primary master is usually a dedicated PTP 
Grandmaster clock device having a time source reference i.e. GPS. 


The Best Master Clock Algorithm (BMC) uses a hierarchical selection algorithm to 
select the active master clock based on properties in the periodic Announce 
messages sent by all master clocks. Clock properties are considered in the following 
order: 


1 = BMC priority 1 

2 class 

3 accuracy 

4 variance 

5 BMC priority 2 

6 unique identifier (tie breaker) 


In the ptp_master_freerun.cfg file the following clock properties are 
configurable (default values are shown): 

# ptp_clock_priority1 128 

# ptp_clock_priority2 128 

Priority1 and priority2 values are in the range 0-255 (0 is the highest priority, 255 the 
lowest). 


CAUTION: When configured as a master clock the Solarflare adapter is using the 
1 \ «UTC timescale. The UTC offset should be set to zero (0) in the sfptpd master config 
file. 
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14 Performance 


14.1 Tickless kernels and the nohz option 


A feature of modern operating systems is that they use a “tickless” kernel which 
aims to reduce power consumption during kernel idle periods, and to increase 
performance: 


e Kernel version 2.6 introduced this feature, stopping the regular timer tick on 
CPU cores which are idle. However, experiments at Solarflare have proven that 
this “tickless” mode degrades PTP performance, producing less consistent 
results than when the kernel always receives periodic timer ticks. 


PTP relies on the ability to accurately change the speed of the system clock by 
very small and precise amounts. The Linux kernel implements this adjustment 
to system clock rate with integer arithmetic, minimizing the error term to the 
target clock rate in every timer tick. However, when the timer tick doesn’t run, 
the error in tracking to the requested clock rate increases, and the system time 
diverges from the clock rate requested. When the system wakes from idle, the 
timer tick runs and the kernel corrects for the error term. 


e Kernel version 3.10 introduced a new mode where only the boot CPU requires 
a tick. Experiments at Solarflare have shown this “full tickless” mode does not 
degrade PTP performance. 


Whether the kernel operates in a tickless mode is configured by the nohz and/or 
nohz_ full boot time options, with the majority of Linux distributions defaulting to 
a tickless kernel. To achieve the highest accuracy with PTP, Solarflare suggest 
configuring the kernel in one of the following ways: 


¢  Donotrun tickless, and so receive timer ticks even when the system is idle. 
This can be achieved by adding "nohz=off" to the kernel boot parameters in 
the /boot/grub/grub. conf file. 

e Run in the “full tickless” mode (requires kernel version 3.10 or later). 


This can be achieved by adding "nohz_full=<cpu_list>" to the kernel boot 
parameters in the /boot/grub/grub. conf file, where <cpu_list> is a list of the 
CPU cores that are to use this mode (and cannot include the boot CPU). 


Alternatively, you can use the CONFIG_NO_HZ_FULL_ALL=y 
kernel configuration parameter to enable this mode for all CPUs except the 
boot CPU. 


You must also ensure that sfptpd runs on a dedicated full tickless CPU, using a 
CPU-affinitizing tool such as taskset. 
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14.2 Accuracy under network load 


To obtain the highest accuracy the PTP protocol requires a network with constant 
latency. Standards such as “PTP boundary clock” and “PTP transparent clock” allow 
network switches to be PTP aware and measure latencies to allow the PTP end 
points to compensate for any variance in switching times for PTP packets. However, 
even with standard non-PTP aware switches, the two stage PTP synchronization 
approach used by the adapter can provide good accuracy under significant network 
load. 


Solarflare has demonstrated slave to master offsets within 100ns ona lightly loaded 
network. However, even under bursty conditions of up to 80% 10G line rate, the 
same network demonstrated slave to master offsets of within 500ns. When the 
bursty condition cleared, the slave to master offsets returned to within 100ns. 


Figure 11 shows PTP accuracy when used in an environment with bursty network 
load of up to 80% line rate. 


Accuracy when under network load 


T T T a 


Time offset from PTP master (ns) 


-200 }- | 
-300 LE i i i i i as 
20% load, 50% load, 80% load, 10% load, 300 byte packets 
300 byte 300 byte MTU sized + 40% load, 300 byte packets, 
packets packets packets 5 seconds on / 2 seconds off 


Figure 11: PTPd under load 
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15 PPS Measurements 


15.1 Asymmetric networks 


Asymmetric networks present a particular problem when attempting to account for 
network latency during PTP offset calculations between master and slave servers. 
PTP assumes symmetry in the network and the PTP protocol is not able to detect 
asymmetry in the network paths between master and slave. 


Asymmetry can be present for a number of reasons including the store and forward 
delay in switches serving asymmetric networks as illustrated in Figure 12. 


Store+forward 


delay 

100 Mbps ———) 10 Gbps 
Master Clock | PTP Slave 
=» 


Store+forward 
delay 


<— 


PPS output PPS input 


Figure 12: Asymmetric network 


The result is that PTP offsets between master and slave will converge, but will be 
wrong by a constant offset from the master equal to half the asymmetry, for 


example 
Transmit time master to slave: Sus 
Transmit time slave to master: lus 


One way delay: (54+1)/2 = 3us 
So 3us is added to the time offsets received from the master clock. 
1PPS will display a mean offset value of 2us (5-3) 
Actual asymmetry should be double this observed value i.e. 4us. 


Measuring and adjusting for asymmetric latency 


Solarflare timestamping adapters support 1PPS input/output interfaces! to allow 
asymmetry in the network to be measured. On a dedicated wire connection 
between master 1PPS output and slave 1PPS input, the master emits a single pulse 
every second. The leading edge of each pulse denotes the exact start of a one 


1. The SFN6322F adapter is factory fitted with 1PPS I/O connectors. Other Solarflare adapters 
require an optional PPS bracket kit and cable assembly (product code SOLR-PPS-DP10G or 
SOLR-PPS-DP40G) available from Solarflare sales channels. 
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second period. When the leading edge of a pulse is detected by the slave adapter, 
firmware on the adapter is able to calculate the offset from its own ‘start of second 
period’. 


e If the initial observed mean 1PPS offset value is a negative value, it means the 
master slave path is slower than the slave>master path, therefore the 
ptp_rx_latency configuration file option on the slave server is used to 
compensate the receive latency. 


e If the initial observed mean 1PPS offset value is a positive value, it means the 
slave>master path is slower than the master>slave path, therefore the 
ptp_tx_latency configuration file option on the slave server is used to 
compensate the transmit latency. 


This 1PPS calibration is only required once when configuring the network and need 
only be performed on one slave server in each network segment which share a 
common network path to the PTP master. There is no need for a permanent 1PPS 
connection to the Solarflare adapter. Refer to 1PPS in practice on page 114. 


15.2 1PPS measurement procedure 


1 = sfptpd should be running between master and slave servers, and should be 
synchronized before the 1PPS value is measured and applied. 


2 The master 1PPS output should be connected to a single slave 1PPS input. 


3 On the slave server, for a short period e.g. 5 minutes, observe the 1PPS mean 
offset value from the pps_off_mean file to identify the mean offset value. 
Refer to 1PPS statistical data on page 116 for instructions on reading the 1PPS 
statistical data files. 


4  Onallslaves on the same network segment, configure sfptpd with knowledge 
of the mean 1PPS offset. 


- If the initial observed 1PPS offset is a negative value, then all subsequent 
offsets should be added as positive values to the ptp_rx_latency option. 
The ptp_tx_latency option in this case should be zero. 


- If the initial observed 1PPS offset is a positive value, then all subsequent 
offsets should be added as positive values to the ptp_tx_latency option. 
The ptp_rx_latency option in this case should be zero. 


5 Continue to observe the 1PPS compensated mean offsets. 


6 Repeat steps 3-5 adding or subtracting the 1PPS mean offset (doubled) value 
each time to the last applied value until the observed 1PPS mean value is as 
close to zero as possible. 
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1PPS asymmetric compensation examples 


Apply to ptp_rx_latency 


Figure 13: 1PPS example 


In the above example the initial observed 1PPS offset is -3400. This value is doubled 
and applied to sfptpd using the ptp_rx_latency parameter as 6800. 


sfptpd is restarted and the next observed 1PPS offset is -300, this value again is 
doubled and applied to the original compensation value (6800 + 600 = 7400). 


sfptpd is restarted and the final observed 1PPS offset is +50 meaning the previous 
compensation value caused sfptpd to over-compensate, so the +50 is doubled and 
subtracted from the previous compensation value (7400 - 100 = 7300). 
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15.3 1PPS in practice 


The following sections demonstrate the effect of applying the 1PPS mean offset 
value to sfptpd. 
1PPS measurements on an asymmetric network 


Figure 14 on page 114 shows the 1PPS offsets observed on an asymmetric network 
consisting of a third party grandmaster clock (1LOOMbps interface) and Solarflare 
10Gbps (slave) adapter connected via a standard network switch. 


1PPS Mean Offsets 
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Figure 14: 1PPS measurement on an asymmetric network 


Identifying the mean offset 


In this particular instance the 1PPS offset is observed for a period of 10 minutes 
before the mean offset value is identified as pps_off_mean: -345@. Refer to 1PPS 
statistical data on page 116 for instructions on reading the 1PPS statistical data. 


Applying the mean offset 


The 1PPS mean offset value should be doubled and applied to sfptpd via the slave 
server configuration file as follows e.g. 


ptp_rx_latency 690@ ptp_tx_latency @ 
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Figure 15 demonstrates 1PPS output after the (doubled) mean offset has been 
applied to sfptpd. 
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Figure 15: 1PPS measurement with offset 


15.4 Solarflare sfptpd 1PPS I/O specification 


e  1PPS-input: SSMB 

- Rising edge active, TTL into 500 
e 1PPS-output: SSMB 

- Rising edge on-time, TTL into 500 
e ~=Pulse Width 

- 200ms high, 800ms low 


15.5 Solarflare 1PPS Connectors and Cabling 


Adapters using 1PPS require the Solarflare 1PPS adapter bracket and cable 
assembly. The PPS IN and PPS OUT connectors on the bracket are SSMB. 


SOLR-PPS-DP1@G SOLR-PPS-DP4@G 


A SSMB to SMA adapter is provided as part of the PPS kit. 
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A 500 coaxial cable or standard SMA 500 connector and coaxial cable assembly is 
required to connect with the PPS pulse source. 


15.6 1PPS statistical data 


1PPS statistical counters and error data is available from the following files: 
/sys/class/net/eth<N>/device/pps_stats/<filename> 

where <filename> is one of the files listed in Table 18. 

Two sets of data are provided in the form of 1PPS offsets (min, max, mean and last) 


and 1PPS periods (min, max, mean and last). All measurements are in nanoseconds. 


Table 18: PPS statistics 


File Description 
pps_off_min Minimum offset value. 
pps_off_max Maximum offset value. 


pps_off_mean Average offset value observed over the last 8 values. 


pps_off_last Most recent offset value. 


pps_per_min Minimum 1PPS period. 


pps_per_max Maximum 1PPS period. 


pps_per_mean Average 1PPS period observed over the last 8 periods. 


pps_per_last Most recent 1PPS period. 


pps_oflow Too many 1PPS values received. Operation is suspended until 
the next sfptpd enable. 


This can occur when a cable is connected or as the result of a 
bad signal or noise on the 1PPS input. 


Re-start the sfptpd processes. 


pps_bad Very bad 1PPS period seen. The 1PPS period measured is too 
long to be a pulse per second i.e. period > 1 second. 


Check the 1PPS input is connected to a genuine 1PPS output. 


Reset statistics counters 


It is possible to reset 1PPS counters in the stats files by writing a‘1’ to the ptp_stats 
file relevant for the Solarflare interface. 


echo 1 > /sys/class/net/eth<N>/device/ptp_stats 
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This section identifies the mechanisms sfptpd uses to generate message logs, state 


logs and statistical data logs. 


State/State Files 


@ 


sfptpd generates state and statistical data in files in the /var/lib/sfptpd directory. 
These files are automatically generated whenever sfptpd is running - no additional 


configuration is required. 


# 1s /var/lib/sfptpd 
freq-correction-000F : 53ff:fe11:2233 
freq-correction-0@@Ff : 53fF:fe44:5566 
freq-correction-system 

interfaces 

ptp-nodes 

state-000f :53ff:fe44:5566 

state-ntp 

state-ptp1 

state-system 
stats-000f: 53ff: fe44:5566 
stats-Q00Ff : 53ff:fe44:5566. json 
stats-ntp 

stats-ntp.json 

stats-ptp1 

stats-ptp1. json 

stats-system 

stats-system. json 

topology 

version 


NOTE: The stats-system file is only generated when the system clock is being 
synchronized to another clock in the host and is controlled by a local clock servo 
process. When running an NTP fallback configuration, the system clock is 
synchronized by NTP -therefore no stats-system clock is created. 


Logfile Update Frequency 


Text format stats files are created for every sync instance and updated every 60 


seconds. 


JSON format files are created only for active sync instances and are updated in 
realtime when events change in the sync-instance. 
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Message Log 
By default sfptpd generates a message event log to stderr. The user can redirect the 
message log to file using the following option in the sfptpd config file. 


message log <path/filename> 


Statistics Log 


By default sfptpd generates a plain text stats log to stdout. The user can redirect the 
stats log to file using the following option in the sfptpd config file. 


stats_log <path/filename> 


The stats_log can also be generated in JSON-lines format - see Machine-Readable 
Real-Time Stats below. 


Statistics Log - Journalctl 


When running sfptpd as a service on Linux system supporting systemd, real-time 
stats_log output can be viewed in the journal: 


# journalctl -fu sfptpd 


PTP Packet Dump 


The sfptpd configuration file ptp_pkt_dump option captures/displays all PTP 
packets sent/received by the sfptpd process. 


\ CAUTION: It is strongly recommended NOT to run with the ptp_pkt_dump option 
enabled for extensive periods as this produces a lot of additional data. This option 
should be used for specific debug purposes only. 


TCPDUMDP Packet Capture 
Packet capture applications such as tcpdump can be useful to identify if messages 
reported missing by sfptpd are actually being received at the PTP interface. 
The following example identifies how to use tcpdump to capture only PTP traffic. 
# tcpdump -i <interface> dst port 319 or dst port 320 [-w <filename.pcap>] 


The interface is the Solarflare adapter interface being used by sfptpd. 


e Keep pcap files as small as possible, but large enough to capture an instance of 
the issue sfptpd is experiencing. 


G) NOTE: tcpdump captures packets sent/received on an interface - but these packets 
may be not be delivered to the sfptpd daemon when they are prevented by other 
factors such as iptables rules, firewalls or reverse-path filters. 


Issue 8 © Copyright 2020 Xilinx, Inc 118 


Solarflare Enhanced PTP User Guide 


SOLARFLARE® 


A XILINX COMPANY Monitoring 


16.2 Machine-Readable Real-Time Stats 


Real-time statistical data are generated in JSON-lines format that is both human- 
readable and machine-readable. The jsonl format files are the same data as 
generated by the sfptpd stats_log plain text file. 


JSON format stats are updated whenever a change occurs in the sync-instance. JSON 
files are produced only for active sync-instances - inactive sync-instances do not 
produce realtime stats. 


G) NOTE: With some standard JSON viewer applications it may be necessary to 
convert the JSON-lines format .jsonl to .json formatted files. 


Enable 


Add or enable the following option in the [general] section of the sfptpd config file: 


json_stats <path><filename>.jsonl 


The stats_log is a JSON-lines formatted file (json!) with one stats log record per line 
and where each record is a JSON object. The following extract is an example: 


{ 

"instance":"ptp1", 

"time": "2017-09-05 15:34:03.702499", 

"clock-master":{"name":"gm"}, 

"clock-slave": 

{ 
"name": "phc@(enp4s@fe/enp4sef1)", 
"time": "2017-09-05 15:34:03.702669635", 
"“primary-interface”: "enp4se@fe" 

}, 

"is-disciplining”: true, 

“in-sync": true, 

“alarms":[], 

"stats": 

{ 
"offset" :7.500000, 
"freg-adj":-1150.850663, 
"“one-way-delay":37.500000, 
"parent-id":"Q00f:53ff:fe43:2740", 
"gm-id":"Q00f:53ff:fe43:2740", 
"active-interface":"enp4sefe", 
"p-term":1.500000, 
"i-term":1149.373163 
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Viewing JSON Format Stats 


The JSON-lines formatted stats_log can be read by any standard web browser JSON 
viewer application. An example JSON viewer HTML application is included with the 
sfptpd distribution: 


/sfptpd-3.2.1.1004/examples/sfptpd_json_parse.html 


The JSON viewer is able to parse the stats_log.jsonl file in real-time, i.e. while sfptpd 
is writing to the file, however it is recommended that the stats_log file is rotated 
periodically to keep files to a manageable size. 


The following is a screen-shot taken from the supplied sfptpd_json_parse.html 
viewer. 


Browse... sfptpd_stats.jsonl You may alter which variables are plotted by changing the contents of the chartInputs variable in the source. 


[ | ptp1.offset [ | servo0.offset | |} servot.offset 


0, 
¥g 5 


The user is able to select measurements to be displayed by the sfptpd_json_parse 
viewer by editing fields in the html file. 
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16.3 Machine-Readable Long-Term Stats 


Long-term statistical data is available in the /var/lib/sfptpd directory where there is 
a stats file for every sync_module or local clock e.g. 


stats-000f :53ff:fe44:5566 
stats-Q00Ff : 53ff:fe44:5566. json 
stats-ntp 

stats-ntp.json 

stats-ptp1 

stats-ptp1. json 

stats-system 

stats-system. json 


Plain text and JSON formatted files are automatically generated by sfptpd and can 
be read by any standard JSON viewer application. Text files and JSON files contain 
exactly the same data. 


Long-term stats files are updated every 60 seconds and retain stats data for up to 3 
weeks. Stats files include counters for all ALL PTP messages sent/received by sfptpd. 


Automatically generated by sfptpd - no configuration is required. The following is an 
extract from the JSON formatted stats file: 


"name": "“announce-pkts-rxed", 
"type": "count", 


"values": [ 
{ 
"period": "minute", 
"period-secs": 60, 
"seq-num": 15, 
"samples": 1169, 
"total": 30, 
"start-time": "2017-09-05 17:30:40", 
"end-time": "2017-09-05 17:31:40" }, 
{ 
"period": "minute", 
"period-secs": 60, 
"seq-num": 14, 
"samples": 1170, 
"total": 30, 
"start-time": "2017-09-05 17:29:40", 
"end-time": "2017-09-05 17:30:40" }, 
{ 


"period": "minute", 

"period-secs": 60, 

"seq-num": 13, 

"samples": 1171, 

Migoyea ils 32), 

"start-time": "2017-09-05 17:28:40", 
"end-time": "2017-@9-05 17:29:40" 


The sequence number identifies successive measurement periods. 
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16.4 Remote Reporting of Real-Time Stats 


The IEEE-1588 draft specification proposes a PTP message-based mechanism for 
application-independent monitoring of the timing information from PTP nodes. 


Solarflare sfptpd supports the draft standard and the sfptpd PTP slave will include 
the optional three data sets TLVs in PTP Signaling messages (message type Oxc) sent 
from the sfptpd slave server to a remote monitoring station. 


e =SLAVE_RX_SYNC_TIMING_DATA TLV 

The slave will report the timestamp for received Sync messages. 

e = SLAVE_RX_SYNC_COMPUTED_DATA TLV 

The slave will report the calculated offset and one-way-delay values. 


e =SLAVE_TX_EVENT_TIMESTAMPS TLV 
The slave will report the timestamp for sent Delay_Request messages. 


For further details of the IEEE draft specification users should refer to the 
specification documentation: 


G) NOTE: The draft IEEE-1588 specification document is not yet published. The sfptpd 
implementation follows the draft proposal, but may be revised to meet any changes 
in the released standard. 


Enable - Monitored Node 


On the PTP slave server to be monitored, the following options should be enabled 
in the PTP generic section of the sfptpd config file: 


# PTP Generic Configuration 
# 


[ptp] 

mon_monitor_address <address of remote monitoring station interface> 
mon_rx_sync_timing data 

mon_rx_sync_computed_data 

mon_tx_event_timestamps 


If address is omitted, PTP Signaling messages from the monitored slave will be 
multicast. 


Enable - Monitoring Station 
A server equipped with a Solarflare PTP enabled adapter can be configured in 
remote monitor mode to received data from all monitored slave servers. 
1 ~=Run sfptpd-v3.2.1. 


2 Create the required sync instances in the general header section: 


[general] 
sync_module ptp1 ptp_all_monitor 
json_remote_monitor /tmp/remote.jsonl 
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In the above example two instances are created. 
e —_[ptp1] is an optional normal PTP slave instance. 
e —_ [ptp_all_ monitor] is an optional monitor only instance. 
e ~The file in which monitored data is to collected is also specified. 
3 _In the PTP generic section: 
[ptp] 
remote_monitor 
With the remote_monitor option enabled, all configured PTP instances are able to 
monitor received PTP signaling messages. 
4 Inthe PTP instance (ptp1) section: 
[ptp1] 
ptp_mode slave 
ptp_domain 10 
interface <interface> 
The [ptp1] instance is a normal PTP slave, but it will also monitor PTP signaling 
messages from domain 10 received on the specified interface. 
5 Inthe PTP instance (ptp_all_monitor) section: 
[ptp_all_monitor] 
ptp_mode monitor 
ptp_domain 127 
interface <interface> 
The parameters in the [ptp_all_ monitor] section identify this as a ptp instance in 
‘monitor’ mode. The instance is assigned a unique ptp_domain value and it will 
monitor PTP signaling messages from all domains on the specified interface. 
ptp_mode monitor is a passive mode. The instance is not a ptp slave and not a ptp 
master. 

Output 


The following is an extract from the real-time stats log emitted from the slave 
reporting the TLVs for a received Sync message. 


{ "rx-event": {"monitor-seq-id": 55, 
"monitor-timestamp": "2017-09-06 13:58:29.911882", 
"node": "Q@00f:53ff:fe21:9bb@.2", 

"“parent-port": "Q00f:53ff:fe43:2740.1", 
"sync-seq": 9817, 

"offset-from-master": 2920.500000, 
"mean-path-delay": 7.500000, 
"sync-ingress-timestamp": 1504702709.911463074 } } 
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The table provides a description of the reported fields: from the rx-event. 


Field Description 
monitor-seq-id Sequence number 
monitor-timestamp Monitored system clock timestamp when the 


rx_event reported is generated 


node Identifies the slave interface clock UUID 

parent-port Identifies the master clock UUID 

sync-seq Sequence number of the received Sync message 
offset-from-master Current offset from master clock 

mean-path-delay Current one-way-delay value 

sync-ingress- Ingress timestamp generated by the adapter on the 
timestamp Monitored server when the Sync message is received 


by the adapter. 


The following is an extract from the real-time stats log emitted from the slave 
reporting the TLVs for a transmitted Delay_Request message. 


{ "tx-event": {"monitor-seq-id": @, 
"monitor-timestamp": "2017-09-06 13:57:43.3324@3", 
"node": "@00f:53ff:fe21:9bb@.2", 

"source-port": "@0@f:53ff:fe21:9bbe.2", 
"message-type": "Delay Req", 

"event-seq-id": 11, 

“egress-timestamp": 1504702403.475124702 } } 


The table provides a description of the reported fields: from the tx-event. 


Field Description 

monitor-seq-id Sequence number 

monitor-timestamp System clock timestamp 

node Identifies the slave interface clock UUID 

source-port Identifies the slave clock port that sent this message 
message-type Delay_Request message 

event-seq-id Sequence ID of the sent Delay_Request 
egress-timestamp Timestamp generated by the adapter when the 


Delay_Request was sent. 
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16.5 Remote Reporting of Slave State 


The standards based slave event monitoring described in Remote Reporting of Real- 
Time Stats above makes no provision for reporting clock alarm and clock status 
information. 


Solarflare sfptpd supports an additional organization extension TLV to provide for 
the reporting of alarms, state and state-change events to also be exported to a 
remote monitoring node. 


When this feature is enabled, the PTP slave server alarms and state changes are sent 
to the remote monitoring node. 


Enable 


On the PTP slave server to be monitored, the following options should be enabled 
in the PTP generic section of the sfptpd config file: 


# PTP Generic Configuration 

# 

[ptp] 

mon_monitor_address <address of remote monitoring node interface> 
mon_slave_status 


When an address is specified, sfptpd will send unicast data. If the address is omitted, 
the sfptpd slave will multicast the data. 


Output 


{ "slave-status": {"monitor-seq-id": Q, 
"monitor-timestamp": "2017-09-06 13:53:18.663545", 
"node": "@00f:53ff:fe41:c700.1", 

"gm-id": "@Q00@:0000:0000:0000.0", 

"state": "DISABLED", 

"bond-changed": false, 

"selected": false, 

"in-sync": false, 

"“msg-alarms": [], 

"alarms": []} } 


{ "slave-status": {"monitor-seq-id": 1, 
"monitor-timestamp": "2017-09-06 13:53:18.663563", 
"node": "@00f:53ff:fe21:9bb@.2", 

"gm-id": "@Q00@:0000:0000:0000.0", 

"state": "LISTENING", 

"bond-changed": false, 

"selected": true, 

"in-sync": false, 

"msg-alarms": [], 

"alarms": []} } 


{ "slave-status": {"monitor-seq-id": 2, 
"monitor-timestamp": "2017-09-06 13:53:21.538047", 
"node": "@00f:53ff:fe21:9bb0.2", 
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"gm-id": "Q000:0000:0000:20000.0", 
"state": "SLAVE", 

"bond-changed": false, 
"selected": true, 

"in-sync": false, 

"msg-alarms": [], 

"alarms": []} } 


16.6 Remote Monitor 


A simple example remote monitor (python) script is included in the 


sfptpd-<version>/examples/monitoring console. py 


16.7 Meinberg NetSync Monitor 


The Meinberg NetSync Monitor capability, supported on Meinberg LANTIME and 
IMS models, allows a remote Monitoring System (MS) to probe downstream PTP 
slave devices using standard PTP messages. NetSync will collect timing data from 
multiple downstream slave devices and compare these to the Meinberg master 
clock timesource so users no longer have to rely on ‘self-reported sync accuracy’ 
from slave devices. An additional MTIE TLV enables the verification of compliance 
with specified MTIE time deviation limits. 


NetSync Monitor uses a mechanism of ‘Reverse PTP’ sending standard PTP 
messages with TLVs attached between the MS and slave devices. Multiple MS can 
be deployed on a network. 


Solarflare sfptpd supports the NetSync feature, recognizes the probe TLVs sent by 
the MS device and will operate the reverse PTP protocol to cooperate with the 
Monitoring Station. 


Enable 


Add the following option to the [general] section of the sfptpd config file: 


mon_meinberg netsync 


Output 


For further details of NetSync Monitor reports and output formats refer to the 
following documentation: 


https://www.meinbergglobal.com/download/docs/sw/english/netsyncmonitor.pdf 
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16.8 Rotate Log Files 


The sfptpdctl logrotate command causes the sfptpd daemon to close and reopen 
the log file,creating the file if it does not already exist. 


G) NOTE: This action does not rename the log file. 


The sfptpdctl logrotate command could be called from the postrotate section of a 
standard Linux logrotate configuration file. For example: 


postrotate 


<path>/sfptpdctl logrotate || true 
endscript 


An alternative action would be to move/rename the current log file before invoking 
the sfptpdctl logrotate command. 
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17 Known Issues and Limitations 


17.1 Firmware upgrade 


CAUTION: The sfptpd daemon must be terminated before upgrading the adapter 
1 \ firmware. Following firmware upgrade the adapter driver should be reloaded. 


e If onload is installed: 


# onload_tool reload 


e — If onload is not installed: 
# modprobe -r sfc 
# modprobe sfc 


CAUTION: The ethtool -t command is disruptive, and should not be used while 
1 \ sfptpd is running. It takes the interface offline, and so interrupts service to it. It 
also resets the adapter clock back to the UTC epoch. 


17.2 PTP and SolarCapture 


When using Solarflare SolarCapture™ to capture packets from an interface also 
being used to send/receive PTP messages, PTP hybrid mode will not function 
correctly when SolarCapture consumes the ARP response messages. This prevents 
the unicast Delay_Request messages being sent from the PTP slave. sfptpd in 
multicast mode is not affected and users are advised to select multicast mode in the 
sfptpd configuration file. 


ptp_network_mode multicast 


Refer to the SolarCapture User Guide configuration options when using 
SolarCapture and sfptpd on the same server. 


17.3 Bonding 


Using the Linux bonding driver, sfptpd supports LACP and active/passive bonding. 
There are limitations to combining PTP and non-PTP ports in the same bond: 


e for LACP bonding, a combination of PTP and non-PTP ports does not work. 


e for active/passive bonding, the quality of convergence depends on whether 
the bond supports timestamping. 
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17.4 LACP 


The PTP protocol cannot work correctly over an LACP bond when the bonded 
interfaces connect to different Boundary Clocks. This is because each Boundary 
Clock has a unique clock identifier. 


The PTP slave can receive sync/followup messages on one link, but it is non- 
deterministic which link it will send the delay-request messages on. When delay- 
request is sent on another link in the bond, delay-response messages returned will 
be from a different clock to the clock identified in the sync/followup messages and 
will be ignored by the PTP slave. 


17.5 Ubuntu - NTP Modes 


When starting sfptpd in one of the supported NTP modes on Ubuntu, it is important 
to setup NTP authentication correctly (see NTP authentication on page 74) and 
important to identify the NTP configuration file being used by the NTP service. 


The usual files will probably exist i.e /etc/ntp.conf and /etc/ntp/keys. 


But the NTP service may be using another ntp.conf file and this prevents sfptpd from 
controlling the local NTP client when the authentication settings have not been 
enabled in the correct NTP configuration file. 


This results is output from sfptpd (stats_log or message_log) similar to the 
following: 


notice: ptp ptpm: ptp ptpm: now in state: PTP_LISTENING 

warning: ntpclient: mode6: failed to set NTP daemon system flags, 
Permission denied 

error: ntp: failed to disable NTP clock control 

critical: failed to create sync module ntp, Permission denied 
critical: couldn't create sync engine thread, Permission denied 


Corrective Action 


1 Use systemctl to identify the NTP configuration file being used by the NTP 
service: 


# systemctl status ntp 


ntp.service - Network Time Service 

Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: 
enabled) 

Active: active (running) since Fri 2019-93-22 12:28:10 GMT; 6min ago 
Docs: man:ntpd(8) 

Process: 14222 ExecStart=/usr/lib/ntp/ntp-systemd-wrapper (code=exited, 
status=@/SUCCESS) 

Main PID: 14230 (ntpd) 

Tasks: 2 (limit: 4915) 

CGroup: /system.slice/ntp.service 

14230 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -c /run/ntp.conf.dhcp - 
u 107:113 


Issue 8 © Copyright 2020 Xilinx, Inc 129 


Solarflare Enhanced PTP User Guide 


SOLARFLARE® 


A XILINX COMPANY Known Issues and Limitations 


In the above example, the NTP service is using the /run/ntp.conf.dhcp 
configuration file. 


2 Edit the file to enable the required NTP authentication: 
keys <path the ntp keys file i.e. /etc/ntp/keys or /etc/ntp.keys> 
trustedkey <value> 
requestkey <value> 


enable mode7 


NOTE: There is no space between ‘mode’ and ‘7’. Ensure the keys file contains 
authentication values identical to those in the sfptpd config file. 


3 ~=Restart the NTP service: 
# systemctl restart ntp 
4 Restart sfptpd. 


Always check the output in the sfptpd messages _ log and stats_log when sfptpd 
starts up - checking for any warning entries or error entries. 
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A How PTP Works 


This section provides a basic description of how the PTP protocol operates between 
a master and a slave server. For a complete description of the PTP protocol refer to 
the IEEE 1588-2008 Standard for a Precision Clock. 


A.1 Message sequence 


The following diagram describes the PTP protocol message sequence which must 
occur for master and slave servers to synchronize. 


Master Clock Slave Clock 


Sync 
message 


Sync Follow up 
message 


Delay request 
message 


Delay Response 
message 


Master Slave messages 


Figure 16: PTP message sequence 
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The Sync message is multicast to all slaves at a fixed interval of between 1 and 64 
messages per second, configurable by the master clock. On most PTP networks a 
sync interval of between 1-4 sync messages per second is sufficient to ensure 
accurate synchronization and increasing the sync interval does not always result in 
greater accuracy of synchronization. When using 1-step synchronization, the Sync 
message contains the time the message was transmitted (T1). The slave generates a 
hardware timestamp (T2) when the message is received. 


The Follow_up message is sent immediately following every Sync by master clocks 
using 2-step synchronization. The Follow_up message contains the actual time the 
preceding Sync message was sent. A master clock using 1-step synchronization does 
not transmit the Follow_up message. 


When the slave has received the Follow_up message (or just Sync message in the 
case of 1-step synchronization) it will generate a Delay_Request message. When 
this message is sent the slave generates and retains a hardware timestamp (T3). 


The master will record the time the Delay_Request is received (T4) and this 
timestamp is then relayed back to the slave in the Delay_Response message. 


Using the timestamp information derived from the message sequence, the slave is 
able to calculate the one-way-delay between slave and master clocks and the time 
offset from the master clock. 


one_way_delay=((T2-T1) + (T4-T3)) / 2 
offset=((T2-T1) - (T4-T3)) / 2 


A.2 Delay-Request interval 


The interval between Delay_Request messages is determined by the master clock. 
A parameter of the Delay_Response message sent from the master is the 
logMessagePeriod. This is a power of 2 value that defines a send window period 
designed to ensure (1) that multiple slaves send at random intervals during the 
period, (2) that the master clock is able to respond to Delay_Requests from multiple 
slaves without queuing these messages. 


window = (2*logMessagePeriod) 
So a logMessagePeriod of 3: 
window = (2%3) = 0-8 second window 


Solarflare sfptpd can override the logMessagePeriod value using the config file 
option ptp_delayreg_interval causing the sfptpd slave to send Delay_Request 
messages at a fixed interval. The ptp_delayreq_interval is also a power of 2 
value. 


Ideally a slave server should sent Delay-Request messages at the same rate as it 
receives Sync/FollowUp messages thus ensuring that the four timestamps: T1-T4 are 
current when calculating the one-way-delay and offset values. Refer to section A.1 
above. 
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A.3 Announce message 


Before a PTP slave server can begin to synchronize with an upstream PTP master 
clock, it must receive PTP Announce messages. Before Announce messages are 
received and the master clock is selected, all other PTP messages received by the 
PTP slave are ignored. 


The Announce messages contains data advertising the master clock type, accuracy 
and priority levels. The Announce message is used by the Best Master Clock 
algorithm to determine the most accurate master clock on a PTP network. 


If a PTP sync module expects to receive an Announce messages, but does not do so, 
it outputs a warning message: 


2016-12-21 11:47:19.652778: warning: failed to receive Announce within 12.0@@ seconds 


Refer to Failed to Receive Announce Messages on page 144. 


A.4 Solarflare sfptpd 2-stage synchronization 


The PTP messages are used by sfptpd to synchronize the adapter clock with the 
master clock. sfptpd runs a second clock servo to synchronize the system clock to 
the adapter clock as illustrated by Figure 17 on page 133. This unique two stage 
synchronization has a number of benefits including: 


e Improved accuracy with the ability to more frequently discipline the system 
clock than is supported by standard PTP masters. 


e = Ability to discipline the system clock to a high precision clock during periods, 
for whatever reason, whereby the upstream PTP master is inaccessible or 
offline. 


system time 


PTP Master 


Figure 17: sfptpd 2-stage synchronization 
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B.1 Using sfptpd under systemd control 


@ 


The following procedure is an example of how to run sfptpd under systemd control 
on RHEL7, CENTOS7 or Ubuntu (15.04 and later). 


Running sfptpd as a service requires a UNIT file so the sfptpd service can be 
managed by systemctl and automatically restarted following server reboot. 


NOTE: sfptpd from version 3.3.0.1007, when installed from the RPM file, will install 
a default systemd UNIT file and default configuration file. For more information 
refer to Using the Binary RPM Package: SF-113122-LS on page 138 below. 


Using the Tarball Package: SF-108910-LS 


Create the UNIT file 


2 C2 


Create the sfptpd.service unit fileas /etc/systemd/system/sfptpd. service 
[Unit] 

Description=sfptpd 

DefaultDependencies=true 

Wants=network-online. target 

After=network-online.target 


[Service] 
ExecStart=/usr/sbin/sfptpd -f/<path>/ptp_slave.cfg 


[Install] 
WantedBy=multi-user.target 


The above example ensures that sfptpd is not started until the network- 
online.target is available to ensure that all network interfaces are configured and in 


an ‘up’ state before sfptpd is started. The sfptpd interface can be specified in the 
ptp_slave.cfg file. 


NOTE: The full path is required to the sfptpd config file. 


NOTE: The sfptpd interface must be configured with an IP address and must be in 
an ‘up’ state before sfptpd can be run. 


NOTE: Run ‘systemctl daemon-reload’ after changes are made to the unit file. 
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Locate the sfptpd binary 


Copy the sfptpd binary into the directory where it will run from: 
/usr/sbin/sfptpd 


Enable required services 


# systemctl enable sfptpd 


Created symlink from /etc/systemd/system/multi-user.target.wants/ 
sfptpd.service to /usr/lib/systemd/system/sfptpd.service. 


e The enable command will create the symbolic links from the systems copy of 
the unit file (usr/lib/systemd/system/sfptpd.service) to the location where 
systemd looks for autostart files (/etc/systemd/system.<target>.target.wants). 


To prevent automatic startup following reboot, disable the service: 
# systemctl disable sfptpd 


This Failed message, when seen after trying the enable command, usually indicates 
a syntax error in the unit file: 


Failed to execute operation: Bad message 


Configure the sfptpd interface 


Users should make sure the sfptpd interface is configured to meet network 
requirements. One method is to create the interface config file which is picked up 
by the network service from the network-scripts directory: 


# cat /etc/sysconfig/network-scripts/ifcfg-enp4sefe 
DEVICE="enp4sefe" 

TYPE="Ethernet" 

BOOTPROTO="none" 

ONBOOT="yes" 

HWADDR="00:0f:53:1a:3b:4c" 

IPADDR="172.11.123.123" 

PREFIX=24 

DEFROUTE="yes" 

UUID="04426332 -5e84-45fd-ad79@-17c8790c99a9" 


Check service status 


Following server reboot - check sfptpd status: 


# systemctl status sfptpd 
a sfptpd.service - sfptpd 

Loaded: loaded (/usr/lib/systemd/system/sfptpd.service; enabled; vendor 
preset: disabled) 

Active: active (running) since Mon 2017-98-07 12:32:25 BST; 14min ago 

Process: 941 ExecStartPre=/bin/sleep 5 (code=exited, status=@/SUCCESS) 
Main PID: 1101 (sfptpd) 

CGroup: /system.slice/sfptpd.service 

aa1101 /usr/sbin/sfptpd -f/tmp/sfptpd-3.0.1.1004.x86_64/config/ 

ptp_slave.cfg 
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Identify startup sequence problems 
If sfptpd fails to start or indicates that the PTP interface is not in an ‘up’ state: 
e check systemctl status for information: 
# systemctl status sfptpd 
e examine the system log for startup sequence/issues: 
# cat /var/log/messages | grep -E ‘enp4s@fe|sfptpd' 
Replace enp4sOf0 with the sfptpd interface. 


sfptpd terminates shortly after startup 


When running under systemd control, sfptpd should not be explicitly run as a 
daemon - that means the following option should NOT be enabled in the sfptpd 
config file. 


# Configure sfptpd to run as a daemon 
# daemon 


If this option is enabled, systemd will identify sfptpd as a daemon not under its 
control and it will be terminated. 


This becomes apparent if the sfptpd service terminates/exits very quickly after 
startup without any apparent cause. 


Realtime sfptpd stats log 


When sfptpd is run as a service with the default stats_log setting of stdout: 
stats_log stdout 
The stats log output can be viewed in realtime from journalctl: 


# journalctl -fu sfptpd 


Using the init.d script 


e —- If required, the sfptpd init.d script should be copied into the following 
directory: 

/etc/rc.d/init.d 

e The default sfptpd.conf file is in directory: 

/etc 

e = All installed files can be located from the RPM list: 

# rpm -qlp sfptpd-3.0.0.1007-1.x86_64.rpm 

/etc/sfptpd. conf 

/usr/lib/systemd/system/sfptpd.service 

/usr/sbin/sfptpd 


/usr/sbin/sfptpdctl 
/usr/share/doc/packages/sfptpd 


Issue 8 © Copyright 2020 Xilinx, Inc 136 


SOLARFLARE® 


A XILINX COMPANY 


Solarflare Enhanced PTP User Guide 


Automatic Startup 


/usr/share/doc/packages/sfptpd/LICENSE 
/usr/share/doc/packages/sfptpd/NTP_COPYRIGHT. html 
/usr/share/doc/packages/sfptpd/PTPD2_COPYRIGHT 
/usr/share/doc/packages/sfptpd/config 
/usr/share/doc/packages/sfptpd/config/default-systemd.cfg 
/usr/share/doc/packages/sfptpd/config/default.cfg 
/usr/share/doc/packages/sfptpd/config/freerun.cfg 
/usr/share/doc/packages/sfptpd/config/many_instances.cfg 
/usr/share/doc/packages/sfptpd/config/ntp. cfg 
/usr/share/doc/packages/sfptpd/config/pps_ slave.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_boundary.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_domain_bridge.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_master_freerun.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_master_ntp.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_slave.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_slave_multiple.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_slave_ntp_fallback.cfg 
/usr/share/doc/packages/sfptpd/examples 
/usr/share/doc/packages/sfptpd/examples/Makefile.sfptpdctl 
/usr/share/doc/packages/sfptpd/examples/README.sfptpdctl 
/usr/share/doc/packages/sfptpd/examples/init.d 
/usr/share/doc/packages/sfptpd/examples/init.d/sfptpd 
/usr/share/doc/packages/sfptpd/examples/monitoring console.py 
/usr/share/doc/packages/sfptpd/examples/sfptpd_json_parse.html 
/usr/share/doc/packages/sfptpd/examples/sfptpd_stats_collectd.py 
/usr/share/doc/packages/sfptpd/examples/sfptpdctl.c 
/usr/share/doc/packages/sfptpd/examples/sfptpdct1l. py 
/usr/share/doc/packages/sfptpd/examples/systemd 
/usr/share/doc/packages/sfptpd/examples/systemd/sfptpd.service 


Configure the sfptpd interface 


Edit the default configuration file - or copy one of the example configuration files to 
the /etc directory and add the sfptpd interface. 


Start the sfptpd service 


# service sfptpd [start|stop|restart] 


Useful systemd commands 


# systemctl status sfptpd 

# systemctl is-active sfptpd 
# systemctl is-enabled sfptpd 
# systemctl is-failed sfptpd 
# systemctl list-units 


list-units only displays units that systemd has attempted to parse and load into 
memory. 


To see all — including the units it did not attempt to parse/load: 
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# systemctl list-unit-files 


Using the Binary RPM Package: SF-113122-LS 
1 Install the RPM 


# rpm -ivh sfptpd-3.3.0.1007-1.x86_64.rpm 
warning: sfptpd-3.3.0.1007-1.x86_64.rpm: Header V4 RSA/SHA256 Signature, 
key ID ca969f38: NOKEY 


Preparing... JHAHHEHHE HH [100% | 
Updating / installing... 
1:sfptpd-3.3.6.1007-1 JHHHHEHHE a [100% | 


sfptpd version 3.3.0.1007 includes and will install a default systemd UNIT file in the 
following directory: 


/usr/lib/systemd/service/sfptpd.service 


A default configuration file is installed at: 
/etc/sfptpd.conf 


The user can edit the UNIT file or configuration file or replace the configuration with 
another - see provided examples 


2 Enable the service 
# systemctl enable sfptpd 


sfptpd can be stopped/started through systemdctl and once enabled will 
automatically start following server reboot. 


3 List RPM installed files 
# rpm -qpl sfptpd-3.3.0.1007-1.x86_64.rpm 


/etc/sfptpd. conf 

/usr/lib/systemd/system/sfptpd.service 

/usr/sbin/sfptpd 

/usr/sbin/sfptpdctl 

/usr/share/doc/packages/sfptpd 
/usr/share/doc/packages/sfptpd/LICENSE 
/usr/share/doc/packages/sfptpd/NTP_COPYRIGHT. html 
/usr/share/doc/packages/sfptpd/PTPD2_COPYRIGHT 
/usr/share/doc/packages/sfptpd/config 
/usr/share/doc/packages/sfptpd/config/default-systemd.cfg 
/usr/share/doc/packages/sfptpd/config/default.cfg 
/usr/share/doc/packages/sfptpd/config/freerun.cfg 
/usr/share/doc/packages/sfptpd/config/many_instances.cfg 
/usr/share/doc/packages/sfptpd/config/ntp. cfg 
/usr/share/doc/packages/sfptpd/config/pps_ slave.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_boundary.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_domain_bridge.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_master_freerun.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_master_ntp.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_slave.cfg 
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/usr/share/doc/packages/sfptpd/config/ptp_slave_multiple.cfg 
/usr/share/doc/packages/sfptpd/config/ptp_slave_ntp_fallback.cfg 
/usr/share/doc/packages/sfptpd/examples 
/usr/share/doc/packages/sfptpd/examples/Makefile.sfptpdctl 
/usr/share/doc/packages/sfptpd/examples/README.sfptpdctl 
/usr/share/doc/packages/sfptpd/examples/init.d 
/usr/share/doc/packages/sfptpd/examples/init.d/sfptpd 
/usr/share/doc/packages/sfptpd/examples/monitoring console.py 
/usr/share/doc/packages/sfptpd/examples/sfptpd_json_parse.html 
/usr/share/doc/packages/sfptpd/examples/sfptpd_stats_collectd.py 
/usr/share/doc/packages/sfptpd/examples/sfptpdctl.c 
/usr/share/doc/packages/sfptpd/examples/sfptpdct1l. py 
/usr/share/doc/packages/sfptpd/examples/systemd 
/usr/share/doc/packages/sfptpd/examples/systemd/sfptpd.service 
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C.1 Description 


The following sections identify diagnostic procedures and sfptpd features useful 
when debugging sfptpd problems and when reporting issues to 
support@solarflare.com. 


sfreport 


sfreport on page 140 

PTP Packet Dump on page 141 

TCPDUMP Packet Capture on page 141 

PTP Alarms on page 141 

Failed to Receive Announce Messages on page 144 
Missing Delay_Response Messages on page 145 
Missing Followup Messages on page 146 
Unexpected PDelay_Request Message on page 147 


Ignored followup, SequencelD doesn't match with last Sync message on 
page 148 


Slave Clock offset by 37 seconds on page 149 
Unexpected Driver Version String 4.0 on page 151 


System clock offset spikes or not in-sync on page 152 


sfreport is a Solarflare diagnostic (perl) script that collects information about the 
host server and installed Solarflare adapters. 


The script will generate a HTML output file which should be returned to Solarflare 
support when reporting an issue. sfreport is part of the Solarflare Linux Diagnostics 
package (SF-108317-LS) available from support@solarflare.com. 


sfreport is non-intrusive, but can be run out-of-hours on production systems: 


# perl sfreport.pl 
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PTP Packet Dump 


The sfptpd configuration file ptp_pkt_dump option captures and displays all PTP 
packets sent/received by the sfptpd process. This is different from tcpdump which 
captures packets sent/received at the adapter interface. 


The ptp_pkt_dump option is useful for: 


e Checking that PTP packets are actually being sent/received by sfptpd. (counters 


for all messages types sent/received are accumulated in the /var/lib/sfptpd/ 
stats-<clock id> files. 


e Debugging badly formatted PTP packets. 


CAUTION: It is strongly recommended NOT to run with the ptp_pkt_dump option 
1 \ enabled for extensive periods as this produces a lot of additional data. This option 
should be used for debug purposes only. 


TCPDUMP Packet Capture 


Packet capture applications such as tcpdump can be useful to: 
e Identify if any PTP messages are actually being received on the PTP interface. 


e Identify if messages reported missing by sfptpd are actually being received at 
the PTP interface. 


e To inspect individual fields in PTP messages. 


The following example shows how to use tcpdump to capture only PTP traffic. 
# tcpdump -i <interface> dst port 319 or dst port 320 [-w <filename.pcap>] 


The interface is the Solarflare adapter interface being used by sfptpd. 


e Keep pcap files as small as possible, but large enough to capture an instance of 
the issue sfptpd is experiencing. 


G) NOTE: tcpdump captures packets sent/received on an interface - but these packets 
may be not be delivered to the sfptpd daemon when they are prevented by other 
factors such as iptables rules, firewalls or reverse-path filters. 


PTP Alarms 


The following list identifies all PTP alarms generated by sfptpd. Active alarms will be 
present in the /var/lib/sfptpd/topology file and in state files within this directory. 


Version 3.3.0.1007 sfptpd will log changes to alarm states in the sfptpd message log. 


no-sync-pkts 


PTP Sync packet(s) are not being received. From a tcpdump pcap file, identify if 
packets are actually being received at the PTP interface. 
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Sync/FollowUp pairs have sequential sequence ID values. If some or all Sync packets 
are missing, the user should check these are being generated by the upstream 
master clock. 


no-follow-ups 


PTP FollowUp packet(s) are not being received. From a tcpdump pcap file, identify if 
packets are actually being received at the PTP interface. 


When the upstream master is using 2-step synchronization it will send a FollowUp 
packet for every Sync packet sent. Sync/FollowUp pairs have sequential sequence ID 
values. If some or all FollowUp packets are missing, the user should check these are 
being generated by the upstream master clock. 


The counters in the /var/lib/sfptpd/stats-<clock id> files will identify the number of 
FollowUp messages received. There should be the same number of FollowUps as 
there are Sync messages received. 


no-delay-resps 


The slave will send periodic Delay-Request messages to the upstream master clock. 
The master clock should respond to each with a Delay-Response message having the 
same sequence ID value as the request. 


From a tcpdump pcap file, identify if packets reported as missing are actually being 
received at the PTP interface. 


If some or all Delay-Request packets are missing, the user should check the master 
clock configuration or consult the master clock vendor. 


If Delay-Response messages, reported as missing, are actually being received on the 
PTP interface, check that the requestingSourcePortidentity and 
requestingSourcePortID values in the response message match the clockidentity and 
sourcePortID values in the corresponding request message. It these values do not 
match, the response messages will be ignored. 


no-pdelay-resps 


When using the peer-to-peer delay method this alarm identifies that PDelay- 
Response messages are not being received for all PDelay-Request messages sent by 
the slave server. 


no-pdelay-resp-follow-ups 

When using the peer-to-peer delay method this alarm identifies that PDelay- 
Response-FollowUp messages are not being received by the sfptpd slave. 
no-tx-timestamps 


Identifies that sfptpd is not able to timestamp outgoing packets. 
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no-rx-timestamps 
Identifies that sfptpd is not able to recover adapter timestamps for received PTP 
packets. 
no-interface 
In a bond, there are no slave interfaces available. 
clock-ctrl-failure 
An instance or clock servo is unable to control a clock and attempts to control it 
result in errors. 
pps-no-signal 
sfptpd is unable to detect any incoming PPS signal. Check that the upstream clock is 
generating the PPS signal. Check that PPS cable is connected to the PPS INPUT on 
the Solarflare adapter. 
pps-seq-num-error 
sfptpd is receiving PPS pulses, but reports missing or unexpected PPS sequence 
numbers. Ensure the driver and firmware being by the adapter are up to date. If the 
issue persists, run the sfreport script and return the HTML output along with the 
sfptpd stats log output to support@solarflare.com 
pps-bad-signal 
sfptpd has detected an invalid PPS period i.e. the measured PPS period is too long 
to be a pulse because it exceeds 1 second. 
no-time-of-day 
sfptpd is not detecting a time-of-day signal. This is normally seen when running 
sfptpd in one of the supported NTP/PPS modes. Time-of-day is the time supplied by 
the NTP server. 
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C.2 Failed to Receive Announce Messages 


Failed to Receive Announce 
message in 12.0 seconds 


Command: tcpdump -i <iface> 
N dst port 319 or dst port 320 


Are PTP messages arriving on 


Restart sfptpd the PTP interface ? 


Check network packet routing 
from Master clock and switches 


Change domain on master 
or slave 


Are master and slave using 
the same PTP domain ? 


Check that a rule does not 
prevent PTP packet 
delivery — or disable the 
services 


Are iptables and/or firewalld 
running ? 


Disable rp_filters on the 

PTP interface and All — or 

add explicit route to the 
upstream clock 


Are reverse path filters 
enabled on the PTP interface 
or ALL ? 


Configure ptp source ipeon the 
Boundary Clock 


Is slave connected to a 
Boundary Clock ? 


Is the BC ptp source ipe 
address configured ? 


Send sfreport output + small pcap 
file to support 
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C.3 Missing Delay_Response Messages 


Failed to Receive DelayResp for 
DelayReq sequence number N 


On startup (HYBRID mode) sfptpd tries 

to send 3 x Delay_Requests as unicast. If 

the master does not support/respond to 

unicast, sfptpd will automatically switch 
to multicast mode. 


Is sequence number > 2 


Check that the master is receiving 
the Request message and is 
generating a Response message. 


Are the reported missing 
messages actually being 
received at the PTP interface 
3 


Investigate where missing 
messages are being dropped in the 
network 


Does the clockldentity and 
SourcePortlD in the Request match 
the requestingSourcePortlidentity 
and requestingSourcePortID in the 
response ? 


Report to Upstream clock vendor 


Use tcpdump to capture small pcap file which includes 
one or more messages which are reported missing. 


Send the following to support@ solarflare.com: 
< The pcap file 

The corresponding sfptpd stats_log file 
< ALL files from the /var/lib/sfptpd directory 


The upstream master clock should respond to every Delay_Request message with a 
corresponding Delay_Response message having the same sequence ID value. 


The requestingSourcePortidentity and requestingSourcePortID values in the 
response message must also match the clockldentity and sourcePortID values in the 
request message - otherwise the response message will be ignored. 


Missing one or two Delay_Response messages should not affect synchronization 
accuracy or precision, but missing many of these messages or missing consecutive 
sequence IDs may harm synchronization of the clocks. 
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C.4 Missing Followup Messages 


Failed to Receive FollowUp for Sync 
sequence number N 


A Sync message should be followed by a 
corresponding FollowUp message 
having the same sequence ID value. 


Are the reported missing 
messages actually being 
eceived at the PTP interface ? 


Consult your PTP vendor/supplier to 
find out why FollowUp messages are not 
being sent. 


Use a tcpdump capture file as proof of 
missing messages. 


Use tcpdump to capture small pcap file which includes 
one or more messages which are reported missing. 


Send the following to support@solarflare.com: 
N__ The pcap file 

The corresponding sfptpd stats_log file 
0 ALL files from the /var/lib/sfptpd directory 


When the upstream master uses 2-step synchronization it should send periodic Sync 
messages. For every Sync there should be FollowUp message having the same 
sequence ID value. 


Missing the occasional FollowUp message should not affect synchronization 
accuracy, but missing many of these messages or missing consecutive sequence IDs 
may harm the synchronization of the clocks. 
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C.5 Unexpected PDelay_Request Message 


Unexpected PDelay_Request in end 


to end mode 


The peer-to-peer delay method is an alternative to the more widely used end-to- 
end delay method. Solarflare sfptpd will support both methods with the default 
being end-to-end. The delay method is configurable in the sfptpd configuration file. 


e The end-to-end delay mechanism: 
Delay Request, Delay _Response 

e The peer-to-peer delay mechanism: 
PDelay Request, PDelay_ Response 


When sfptpd is using the end-to-end delay method, it will generate the 
“Unexpected...” warning if a PDelay_Request message is received. 


When sfptpd is using the peer-to-peer delay method, it will generate the 
“Unexpected...” warning if a Delay_Request message is received. 


When peer-to-peer working is used in the network, every PTP node must support 
and use it. 


Mixing end-to-end with peer-to-peer methods is not supported in any PTP domain. 
All PTP nodes within a PTP domain must use the same method. 


When peer-to-peer delay is used every node in the PTP network sends both the 
PDelay_Request and PDelay_Response messages to the neighbouring PTP node. 
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C.6 Ignored followup, SequencelD doesn't match with last Sync 
message 


Ignored followup, SequencelD doesn't match with 


last Sync message, expected <N>, got <N-1> 


Using 2-step synchronization, an upstream master clock will send periodic Sync 
messages and corresponding FollowUp messages to the downstream slave. Sync 
and FollowUp message pairs have the same sequence ID value. 


Sync/FollowUp message pairs should be received by the PTP slave node in sequence 
i.e. 


Receive Sync (seq N), 
Receive FollowUp (seq N) 
Receive Sync (seq N+1) 
Receive FollowUp (seq N+1) 


Under extreme congestion conditions, FollowUp messages may be delayed, lost or 
dropped in the Network. 


The above message is generated by sfptpd if it has received a FollowUp (seq N) after 
it has already received a Sync with (seq N+1) and so it expecting the corresponding 
FollowUp (seq N+1). 


These rare events are indicative of extreme network congestion — which is delaying 
messages, the effects of which will probably be evident in other network 
applications. 


Use tcpdump to capture small pcap file which includes one or more messages which 
are reported out of sequence. 


Send the following to support@solarflare.com: 
e The pcap file 

e The corresponding sfptpd stats_log file 

e = ALL files from the /var/lib/sfptpd directory 
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C.7 Slave Clock offset by 37 seconds 


Slave clock is offset by 37 seconds 


Is the Current UTC offset Capture small pcap file 
value set to 37 in the containing a few Announce 


Ss the PTP_UTC_REASONABL 
flag set in the recevied 
Announce messages ? 


Announce messages ? messages 


Send the following to 
support@solarflare.com 


The pcap file 

The sfptpd config file 

ALL files from the /var/lib/ 
sfptpd directory 


Is the Current UTC offset 
value set to 37 in the 
Announce messages ? 


Set the sfptpd 
ptp_utc_valid_handling to 


ignore in the sfptpd config file. 


Fix the upstream master clock 


PTP Master clocks use the atomic timescale (TAI). Servers in business networks use 
the UTC timescale (UTC). 


The difference between the two timescales is the UTC offset - currently 37 seconds. 
This increments by one second whenever a leap second occurs. 


UTC time = (TAI time - UTC offset) 


G) NOTE: When sfptpd is used as a master clock it uses UTC time - ensure the 
ptp_utc_offset option in the sfptpd master config file is set to 0: 
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If the sfptpd slave clocks are observed to be ~37 seconds offset from the master 
clock, the user should collect a small tcodump pcap file and examine the UTC offset 
values in the received Announce messages. 


A pcap file allows the user to examine the fields of the received PTP Announce 


messages: 
flags: @x@23c 
@... staats 
.@.. resend 
Oe. coud eatiats 
.@.. aacae 
alias axe auarte 
a: erarava Gececace 
alas“ tanlerists 
ice “axgters 
a Les 
5 . aye yes 
0 


originCurrentUTCOffset: 


PTP_SECURITY: False 

PTP profile Specific 2: False 
PTP profile Specific 1: False 
PTP_UNICAST: False 
PTP_TWO_STEP: True 
PTP_ALTERNATE_MASTER: False 
FREQUENCY_TRACEABLE: True 
TIME_TRACEABLE: True 
PTP_TIMESCALE: True 
PTP_UTC_REASONABLE: True 
PTP_LI_59: False 

PTP_LI_61: False 
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C.8 Unexpected Driver Version String 4.0 


unexpected driver version string, 
4.0 


Upgrade the adapter driver 


This error message may be observed in the sfptpd stats_log/message_log during 
sfptpd startup. 


The Linux ‘in-tree’ driver is the Solarflare adapter driver distributed with the Linux 
OS. This is normally the 4.0 or 4.1 version driver. 


This driver does not support the PTP features required by sfptpd. 


The driver should be upgraded to a later version from the Solarflare download portal 
at support@solarflare.com. 


The driver being used by the Solarflare adapter can be identified using the following 
command: 


# ethtool -i <interface> 

driver: sfc 

version: 4.0 

firmware-version: 4.7.1.1001 rx1 tx1 
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C.9 System clock offset spikes or not in-sync 
When observation of the stats_log or state files identifies that the system clock has 
any of the following symptoms: 
e —_ periodic spikes in offset value 
e offset value constantly much greater than ~100ns from the LRC 
e — offset value behaving erratically 
e the in-sync flag value is never 1 


It is possible that the system clock is being adjusted, probed or otherwise being 
interfered with by a process other than sfptpd. 


Another known cause of this is an unstable TSC clock and this will be reported in the 
kernel logs. 


Processes known to adjust the system clock 


Centrify 


This is Windows Active Directory integration for Linux systems providing centralized 
identification and access management to servers. 


To prevent Centrify from adjusting the system clock, disable sntp in the Active 
Directory configuration using the following setting: 


FILE: /etc/centrifydc/centrifydc.conf/ 
PARAMETER: adclient.sntp.enabled false 


A restart of the Centrify/Active Directory service is required following the change. 


Quest 


Another product extending Active Directory authentication services across Linux 
platforms. 


PBIS 


PowerBroker Identify Services is a further product offering Windows Active 
Directory authentication services to Linux platforms. 


systemd-timesyncd 


This is a daemon that implements an SNTP client which achieves a low quality time 
synchronization of the system clock. The SNTP client can step and adjust the system 
clock. 
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systemd-timedated 


This is a service that may be used to change the system clock and timezone, as well 
as to enable/disable NTP time synchronization. This service is activated on request 
and is self-terminating when unused. 


Chronyd 


Chronyd is a deamon implementation of NTP. It can be used to synchronize the 
system clock with NTP servers or reference clocks. 


phc2sys 


phc2sys is a Linux program which is able to synchronize two clocks in a system. 
Typically this is used to synchronize the system clock with a PTP hardware clock. 


NTP 


When running sfptpd, the NTP service should only be enabled when this service is 
required by the sfptpd mode i.e. when running in NTP/PPS mode or an NTP fallback 
mode. 


At startup, sfptpd will warn the user and exit if the NTP service is running, but is not 
required by the sfptpd mode being used. 


tuned - Profiles 


Some tuned profiles may activate any of the services listed above, or other services 
which interact with the system clock. Profiles are normally stored in the /usr/lib/ 
tuned directory. 


How to resolve 


1 _—Itis useful to Identify if the offset value spikes periodically or randomly. This can 
be done by plotting the system clock offset value against the sfptpd runtime 
using the system clock output lines from the sfptpd stats_log. 


2 The stats_log output is updated every second, so over a 60 second runtime 
period there will be ~60 output lines for each active sync instance and for the 
system clock. 


3 Plotting the system clock values over a period of time e.g. 12 hours or 24 hours, 
should be sufficient to identify offset outliers and spike patterns. 


4  Runasystemtap script to identify if any user-level process is periodically 
adjusting or interfering with the system clock. 
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An example system tap script (sysclock_monitor.stp): 


probe nd_syscall.adjtimex, nd_syscall.compat_adjtimex ? 


- printf("1. %s (pid=%d) %s\n", execname(),pid(),pp()) 
— nd_syscall.clock_adjtime ? 

- printf("2. %s (pid=%d) %s\n", execname(),pid(),pp()) 
— nd_syscall.settimeofday ? 

- printf("3. %s (pid=%d) %s\n", execname(),pid(),pp()) 
— nd_syscall.stime ? 

- printf("4. %s (pid=%d) %s\n", execname(),pid(),pp()) 
— nd_syscall.clock_settime ? 

: printf("5. %s (pid=%d) %s\n", execname(),pid(),pp()) 
} 


The script file should be made executable and can then be run on the system (with 
or without sfptpd running): 


# stap -v sysclock_monitor.stp 


Pass 1: parsed user script and 93 library script(s) using 201532virt/ 
29960res/3064shr/27464data kb, in 11@usr/@sys/12@real ms. 

Pass 2: analyzed script: 6 probe(s), 3 function(s), 25 embed(s), @ 
global(s) using 20518@virt/32264res/4012shr/28652data kb, in 1@usr/4@sys/ 
51real ms. 

Pass 3: translated to C into "/tmp/stap8mqNe9/ 
stap_db5e@4ccc51fad4e17bc414fdb1c3685_15211 src.c" using 20518@virt/ 
32500res/4232shr/28652data kb, in Ousr/@sys/@real ms. 

Pass 4: compiled C into "stap_db5e04ccc51fad4e17bc414fdb1c3685_15211.ko" 
in 720usr/13@sys/917real ms. 

Pass 5: starting run. 


Example output when sfptpd and chronyd are running: 

sfptpd (pid=22803) kprobe.function("sys_adjtimex")? 
sfptpd (pid=22803) kprobe.function("sys_adjtimex")? 
sfptpd (pid=22803) kprobe.function("sys_adjtimex")? 
chronyd (pid=1534) kprobe.function("sys_adjtimex")? 
chronyd (pid=1534) kprobe.function("sys_adjtimex")? 
chronyd (pid=1534) kprobe.function("sys_adjtimex")? 


PRPPRPRPR 
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The following script will identify any user-level process trying to adjust the system 
clock, but will not report sfptpd: 


probe nd_syscall.adjtimex, nd_syscall.compat_adjtimex ? 


{ 
if (execname() != "sfptpd"){ 
printf("1. %s (pid=%d) %s\n", execname(),pid(),pp()) 
exit() 
} 
} 
probe nd_syscall.clock_adjtime ? 
{ 
if (execname() != "sfptpd"){ 
printf("2. %s (pid=%d) %s\n", execname(),pid(),pp()) 
exit() 
} 
i 
probe nd_syscall.settimeofday ? 
{ 
if (execname() != "sfptpd"){ 
printf("3. %s (pid=%d) %s\n", execname(),pid(),pp()) 
exit() 
f 
} 
probe nd_syscall.stime ? 
{ 
if (execname() != "sfptpd"){ 
printf("4. %s (pid=%d) %s\n", execname(),pid(),pp()) 
exit() 
} 
} 
probe nd_syscall.clock_settime ? 
{ 
if (execname() != "sfptpd"){ 
printf("5. %s (pid=%d) %s\n", execname(),pid(),pp()) 
exit() 
} 
} 


If there is no output from running this example system tap script, then only the 
sfptpd process is adjusting the system clock. 
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C.10 NTP: ntpdc: failed to get system info from NTP daemon 


This error is normally observed when attempting to start sfptpd in one of the 
supported PTP and NTP/NTP fallback modes: 


warning: ntpdc: failed to get system info from NTP daemon, Connection 
timed out 

error: timed-out retrieving NTP system info. Is ntpdate running? 
critical: failed to create ntp module, Connection timed out 


The error occurs because on startup, sfptpd will attempt to communicate with 
ntpdc on the local loopback interface. 


1 = Check that a loopback interface config file exists in /etc/sysconfig/network- 
scripts/ifcfg-lo. The file would normally contain the following lines: 


DEVICE=lo 

IPADDR=127.0.0.1 

NETMASK=255.0.0.0 

NETWORK=127.0.0.0 

# If you're having problems with gated making 127.0.0.0/8 a martian, 

# you can change this to something else (255.255.255.255, for example) 
BROADCAST=127.255.255.255 

ONBOOT=yes 

NAME=loopback 


2 = Check that the following line is enabled in the /etc/ntp.conf file: 


# Permit all access over the loopback interface. This could 
# be tightened as well, but to do so would effect some of 

# the administrative functions. 

restrict 127.0.0.1 


This could also be a different form: 


interface listen 127.0.0.1 


3. Onsome OSs it may be necessary to explicitly enable NTP mode 7 to allow 
communication with ntpdc/ntpd. This is done by adding the following line 
anywhere in the /etc/ntp.conf file 


enable mode7 


G) NOTE: There is no space between mode and 7. 
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C.11 NTP: ntpclient: failed to set NTP daemon system flags 


This error is normally observed when attempting to start sfptpd in one of the 
supported PTP and NTP/NTP fallback modes: 


warning: ntpclient: mode7: failed to set NTP daemon system flags, 
Permission denied 

error: ntp: failed to disable NTP clock control 

critical: failed to create sync module ntp, Permission denied 
critical: couldn't create sync engine thread, Permission denied 


The error occurs when sfptpd is prevented from communicating with and 
controlling the local ntpclient on the server. It fails because NTP authentication is 
either not configured or is not consistent in all required files. 


NTP symmetric authentication should be configured in the following files: 
e —_ the sfptpd config file 
e = the /etc/ntp.conf file 
e = the /etc/ntp/keys file 


To resolve this - setup NTP symmetric authentication as described in NTP 
authentication on page 74. 


On some OSs it may be necessary to explicitly enable NTP mode 7 to allow 
communication with the ntpclient. This is done by adding the following line 
anywhere in the /etc/ntp.conf file 


enable mode7 


G) NOTE: There is no space between mode and 7. 
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